Jul 28 17:10:10.447197 ip-10-0-136-111 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:10:10.851776 ip-10-0-136-111 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:10.851776 ip-10-0-136-111 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:10:10.851776 ip-10-0-136-111 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:10.851776 ip-10-0-136-111 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:10:10.851776 ip-10-0-136-111 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:10:10.852519 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.851842    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:10:10.858988 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.858963    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:10.858988 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.858984    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:10.858988 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.858988    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:10.858988 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.858992    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.858995    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.858999    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859001    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859004    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859008    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859011    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859014    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859016    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859019    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859022    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859024    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859027    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859030    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859032    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859035    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859038    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859052    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859055    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859058    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:10.859117 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859060    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859063    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859066    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859069    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859072    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859076    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859080    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859083    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859086    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859089    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859092    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859094    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859097    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859100    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859103    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859106    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859109    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859112    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859114    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:10.859641 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859117    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859120    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859122    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859125    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859128    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859130    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859133    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859136    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859138    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859140    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859143    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859146    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859149    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859152    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859154    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859156    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859159    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859162    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859164    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859167    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:10.860192 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859169    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859172    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859174    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859177    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859179    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859182    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859185    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859188    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859191    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859194    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859197    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859201    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859204    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859207    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859210    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859212    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859215    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859218    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859221    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859227    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:10.860829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859231    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859234    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859236    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859239    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859242    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859245    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859247    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859250    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859252    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859255    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859257    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859260    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859262    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859265    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859268    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859271    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:10.861670 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.859274    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861759    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861774    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861781    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861787    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861792    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861797    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861802    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861806    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861811    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861815    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861820    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861824    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861828    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861833    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861837    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861842    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861846    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861850    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861855    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:10.862368 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861860    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861864    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861869    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861873    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861897    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861902    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861906    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861910    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861915    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861919    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861923    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861929    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861933    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861938    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861943    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861951    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861959    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861965    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861971    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:10.863241 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861976    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861981    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861986    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861991    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.861996    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862000    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862005    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862010    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862015    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862020    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862024    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862029    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862033    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862038    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862042    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862046    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862050    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862055    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862059    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862063    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:10.864088 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862067    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862071    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862076    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862080    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862084    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862088    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862093    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862097    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862102    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862106    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862110    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862114    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862118    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862122    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862126    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862130    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862134    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862138    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862143    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862154    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:10.865013 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862158    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862162    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862167    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862172    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862176    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862182    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862190    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862195    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862200    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862204    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862209    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862213    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862218    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862223    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862227    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862231    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862237    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862241    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862246    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:10.865522 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862250    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862255    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862909    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862918    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862923    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862928    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862933    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862937    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862941    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862945    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862950    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862954    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862959    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862963    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862967    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862971    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862975    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862980    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862986    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862991    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:10.866101 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.862996    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863000    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863005    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863009    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863013    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863018    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863022    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863026    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863030    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863035    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863039    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863044    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863049    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863053    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863057    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863062    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863066    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863070    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863075    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863080    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:10.866757 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863084    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863088    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863092    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863097    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863104    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863109    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863114    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863119    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863123    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863127    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863132    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863137    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863141    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863145    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863149    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863153    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863158    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863162    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863166    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863170    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:10.867348 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863174    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863178    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863182    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863187    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863192    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863197    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863201    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863205    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863210    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863214    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863218    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863222    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863226    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863231    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863235    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863239    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863245    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863252    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863257    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:10.867848 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863261    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863270    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863275    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863280    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863284    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863290    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863295    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863299    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863303    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863307    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863312    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863316    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863320    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863324    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863328    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863333    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863337    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863341    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863346    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863350    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:10.868421 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863354    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.863359    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863466    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863478    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863488    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863495    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863502    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863507    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863514    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863520    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863526    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863531    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863536    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863543    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863548    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863553    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863557    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863562    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863569    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863574    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863582    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863586    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863592    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863597    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:10:10.868921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863602    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863615    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863620    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863625    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863639    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863644    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863649    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863654    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863659    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863664    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863670    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863675    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863680    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863685    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863690    2579 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863694    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863701    2579 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863706    2579 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863711    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863716    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863720    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863728    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863734    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863739    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863744    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:10:10.869521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863748    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863753    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863768    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863774    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863779    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863784    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863789    2579 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863795    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863800    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863805    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863809    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863815    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863820    2579 flags.go:64] FLAG: --help="false"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863825    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-136-111.ec2.internal"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863830    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863835    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863840    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863845    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863851    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863855    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863860    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863865    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863869    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:10:10.870149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863896    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863902    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863906    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863911    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863915    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863923    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863928    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863932    2579 flags.go:64] FLAG: --lock-file=""
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863937    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863941    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863946    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863955    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863961    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863966    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863971    2579 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863976    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863982    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863986    2579 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863991    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.863998    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864003    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864010    2579 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864015    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864020    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864025    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:10:10.870750 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864029    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864034    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864039    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864044    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864056    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864061    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864066    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864071    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864076    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864084    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864088    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864093    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864099    2579 flags.go:64] FLAG: --port="10250"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864104    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864108    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03f178fb47d653ff6"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864113    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864118    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864123    2579 flags.go:64] FLAG: --register-node="true"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864128    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864134    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864140    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864145    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864149    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864156    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864160    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:10:10.871544 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864165    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864170    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864175    2579 flags.go:64] FLAG: --runonce="false"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864181    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864186    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864191    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864196    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864200    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864205    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864210    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864215    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864220    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864225    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864230    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864234    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864239    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864244    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864248    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864257    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864263    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864268    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864275    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864280    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864284    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864289    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:10:10.872183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864294    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864299    2579 flags.go:64] FLAG: --v="2"
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864306    2579 flags.go:64] FLAG: --version="false"
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864313    2579 flags.go:64] FLAG: --vmodule=""
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864319    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.864324    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864534    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864542    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864547    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864552    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864556    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864561    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864566    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864570    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864575    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864579    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864583    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864588    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864592    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864596    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864601    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864605    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:10.872794 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864609    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864618    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864622    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864626    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864633    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864637    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864642    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864646    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864650    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864654    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864659    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864663    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864667    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864672    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864677    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864681    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864688    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864693    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864698    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:10.873358 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864703    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864708    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864713    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864717    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864723    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864728    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864732    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864737    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864742    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864746    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864751    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864756    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864763    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864770    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864776    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864780    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864785    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864790    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864795    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:10.873829 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864799    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864803    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864808    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864812    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864816    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864820    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864824    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864828    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864833    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864838    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864842    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864846    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864851    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864855    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864861    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864866    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864871    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864892    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864897    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864902    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:10.874406 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864906    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864911    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864916    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864920    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864925    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864930    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864935    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864941    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864946    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864950    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864955    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864960    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864964    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864968    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864972    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864977    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864981    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864985    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864989    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:10.874942 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864994    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.864998    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.865002    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.865007    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.865011    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.865015    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.865646    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.872768    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.872788    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.872810    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.872816    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872874    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872893    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872896    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:10.875408 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872900    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872903    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872906    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872909    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872912    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872917    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872921    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872924    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872927    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872930    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872932    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872935    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872938    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872941    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872944    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872947    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872950    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872952    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872955    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:10.875762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872957    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872960    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872962    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872965    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872968    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872971    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872973    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872977    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872980    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872982    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872985    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872987    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872990    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872993    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872995    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.872998    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873000    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873003    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873005    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873008    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:10.876250 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873010    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873013    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873015    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873018    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873021    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873024    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873026    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873029    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873032    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873035    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873037    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873054    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873057    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873060    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873063    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873066    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873069    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873072    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873075    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873078    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:10.876732 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873082    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873085    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873087    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873090    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873103    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873107    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873111    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873114    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873116    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873119    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873122    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873125    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873128    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873130    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873133    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873135    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873138    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873140    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873143    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:10.877263 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873146    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873149    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873151    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873154    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873157    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873159    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873162    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873164    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873167    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873170    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873174    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873178    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873182    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873185    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873190    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873193    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873195    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:10.877721 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873198    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.873203    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873325    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873329    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873332    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873335    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873338    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873340    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873344    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873346    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873349    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873351    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873354    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873357    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873359    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:10:10.878329 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873362    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873365    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873368    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873370    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873373    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873376    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873378    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873381    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873384    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873386    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873389    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873391    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873394    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873397    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873399    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873403    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873405    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873408    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873411    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873413    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:10:10.878690 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873416    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873419    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873421    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873424    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873426    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873429    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873432    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873434    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873437    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873439    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873441    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873444    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873446    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873449    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873452    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873454    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873458    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873462    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873464    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:10:10.879209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873467    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873470    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873474    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873478    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873481    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873484    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873487    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873490    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873492    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873495    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873498    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873501    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873503    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873506    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873509    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873511    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873514    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873516    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873519    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873521    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:10:10.879664 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873524    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873527    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873529    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873532    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873534    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873537    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873539    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873541    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873544    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873547    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873550    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873552    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873556    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873559    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873562    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873564    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873568    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873570    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873573    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:10:10.880161 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873575    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873578    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873581    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873584    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873586    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873589    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873592    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:10.873594    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.873600    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.873731    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.875891    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.877390    2579 server.go:1019] "Starting client certificate rotation"
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.877509    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:10.880618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.877546    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:10:10.880953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.880013    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:10.883064 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.883041    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:10:10.897714 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.897688    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:10:10.899874 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.899856    2579 log.go:25] "Validated CRI v1 image API"
Jul 28 17:10:10.900646 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.900629    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:10:10.904578 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.904557    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 86116e27-d051-4556-b54f-9e39d31d7af5:/dev/nvme0n1p3 8e7471e7-620c-471c-8016-d1f8d92eba3b:/dev/nvme0n1p4]
Jul 28 17:10:10.904650 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.904578    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:10:10.908484 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.908466    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:10.909916 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.909795    2579 manager.go:216] Machine: {Timestamp:2026-07-28 17:10:10.908264502 +0000 UTC m=+0.343663309 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098475 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26afd603820bf462cf7098447d7294 SystemUUID:ec26afd6-0382-0bf4-62cf-7098447d7294 BootID:9cac52ea-fafa-4143-9c2b-06e9f617ab20 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:40:09:9e:55:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:40:09:9e:55:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:2c:74:6b:2a:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:10:10.909916 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.909913    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:10:10.910031 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.909999    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:10:10.913045 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.913014    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:10:10.913187 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.913046    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-111.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:10:10.913771 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.913760    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:10:10.913804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.913773    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:10:10.913804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.913780    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:10.914507 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.914497    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:10:10.915970 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.915960    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:10:10.919561 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.919549    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:10.919690 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.919680    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:10:10.919736 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.919728    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:10:10.919778 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.919750    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:10:10.919778 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.919770    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:10:10.919872 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.919781    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:10:10.919872 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.919791    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:10:10.920223 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.920210    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:10.920277 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.920229    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:10:10.922586 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.922570    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:10:10.924640 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.924626    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:10:10.924689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.924657    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:10:10.926376 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926359    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:10:10.926376 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926377    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:10:10.926515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926383    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:10:10.926515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926388    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:10:10.926515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926399    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:10:10.926515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926408    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:10:10.926515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926417    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:10:10.926515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926423    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:10:10.926515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926429    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:10:10.926515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926435    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:10:10.926515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926453    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:10:10.926515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.926458    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:10:10.927307 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.927297    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:10:10.927307 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.927307    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:10:10.930596 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.930571    2579 server.go:1268] "Started kubelet"
Jul 28 17:10:10.930715 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.930700    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-111.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:10:10.930767 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.930642    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:10:10.930905 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.930867    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:10:10.930972 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.930925    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:10:10.931495 ip-10-0-136-111 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:10:10.931767 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:10.931582    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:10:10.931767 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:10.931584    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:10:10.934204 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.934169    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:10:10.935118 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.935099    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:10:10.937450 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:10.937426    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:10:10.942447 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:10.941447    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-111.ec2.internal.18c6833f78b8eb07  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-111.ec2.internal,UID:ip-10-0-136-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-111.ec2.internal,},FirstTimestamp:2026-07-28 17:10:10.930543367 +0000 UTC m=+0.365942173,LastTimestamp:2026-07-28 17:10:10.930543367 +0000 UTC m=+0.365942173,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-111.ec2.internal,}"
Jul 28 17:10:10.942563 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.942545    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:10:10.942563 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.942560    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:10.942701 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:10.942624    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:10.943222 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943206    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:10:10.943222 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943208    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:10:10.943222 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943233    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:10:10.943379 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943306    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:10:10.943379 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943315    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:10:10.943608 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943594    2579 factory.go:55] Registering systemd factory
Jul 28 17:10:10.943608 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943611    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:10:10.943831 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943817    2579 factory.go:153] Registering CRI-O factory
Jul 28 17:10:10.943890 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943834    2579 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:10:10.943925 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943900    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:10:10.943925 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943923    2579 factory.go:103] Registering Raw factory
Jul 28 17:10:10.943995 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.943937    2579 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:10:10.944335 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.944323    2579 manager.go:349] Starting recovery of all containers
Jul 28 17:10:10.946825 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:10.946794    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:10:10.946944 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:10.946824    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-111.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:10:10.953262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.953241    2579 manager.go:354] Recovery completed
Jul 28 17:10:10.957984 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.957972    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:10.959832 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.959816    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:10.959921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.959847    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:10.959921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.959861    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:10.960633 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.960618    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:10:10.960633 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.960628    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:10.960633 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.960630    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:10:10.960822 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.960649    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:10:10.961340 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.961326    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:10.961407 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.961351    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:10.961407 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.961362    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:10.962376 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:10.962276    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-111.ec2.internal.18c6833f7a77d3b7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-111.ec2.internal,UID:ip-10-0-136-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-111.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-111.ec2.internal,},FirstTimestamp:2026-07-28 17:10:10.959831991 +0000 UTC m=+0.395230796,LastTimestamp:2026-07-28 17:10:10.959831991 +0000 UTC m=+0.395230796,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-111.ec2.internal,}"
Jul 28 17:10:10.962712 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.962701    2579 policy_none.go:49] "None policy: Start"
Jul 28 17:10:10.962762 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.962716    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:10:10.962762 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.962743    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:10:10.964392 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.964380    2579 policy_none.go:47] "Start"
Jul 28 17:10:10.969626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.969606    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h6kg7"
Jul 28 17:10:10.971673 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:10.971603    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-111.ec2.internal.18c6833f7a782aa1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-111.ec2.internal,UID:ip-10-0-136-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-111.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-111.ec2.internal,},FirstTimestamp:2026-07-28 17:10:10.959854241 +0000 UTC m=+0.395253046,LastTimestamp:2026-07-28 17:10:10.959854241 +0000 UTC m=+0.395253046,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-111.ec2.internal,}"
Jul 28 17:10:10.975776 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:10.975756    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-h6kg7"
Jul 28 17:10:10.982149 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:10.982056    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-111.ec2.internal.18c6833f7a7855ea  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-111.ec2.internal,UID:ip-10-0-136-111.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-111.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-111.ec2.internal,},FirstTimestamp:2026-07-28 17:10:10.959865322 +0000 UTC m=+0.395264127,LastTimestamp:2026-07-28 17:10:10.959865322 +0000 UTC m=+0.395264127,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-111.ec2.internal,}"
Jul 28 17:10:11.001596 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.001577    2579 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:10:11.001696 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.001611    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:10:11.001696 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.001622    2579 server.go:87] "Starting device plugin registration server"
Jul 28 17:10:11.001963 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.001950    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:10:11.002023 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.001966    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:10:11.002108 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.002093    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:10:11.002235 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.002225    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:10:11.002318 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.002306    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:10:11.002751 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.002729    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:10:11.002843 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.002782    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:11.024828 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.024808    2579 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2613 into the system container \"/system.slice\": <nil>"
Jul 28 17:10:11.102330 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.102120    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:11.103304 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.103289    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:11.103389 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.103318    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:11.103389 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.103328    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:11.103389 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.103355    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.105842 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.105820    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:10:11.107054 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.107030    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:10:11.107135 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.107077    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:10:11.107135 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.107101    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:10:11.107238 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.107141    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:10:11.109283 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.109265    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:11.115624 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.115607    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.115721 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.115632    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-111.ec2.internal\": node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:11.156850 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.156819    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:11.208005 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.207961    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal"]
Jul 28 17:10:11.208100 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.208050    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:11.209590 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.209567    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:11.209718 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.209603    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:11.209718 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.209622    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:11.211333 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.211319    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:11.211473 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.211457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.211540 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.211494    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:11.212928 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.212915    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:11.213007 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.212941    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:11.213007 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.212950    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:11.213007 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.212915    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:11.213122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.213024    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:11.213122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.213038    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:11.214933 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.214920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.214983 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.214946    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:10:11.215994 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.215977    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:10:11.216077 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.216009    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:10:11.216077 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.216023    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:10:11.239658 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.239636    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-111.ec2.internal\" not found" node="ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.243918 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.243903    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-111.ec2.internal\" not found" node="ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.244965 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.244949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb470419a847df4b595db3785a098356-config\") pod \"kube-apiserver-proxy-ip-10-0-136-111.ec2.internal\" (UID: \"cb470419a847df4b595db3785a098356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.245025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.244975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0d871ec4bfc496edfc43fbc77317825-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal\" (UID: \"e0d871ec4bfc496edfc43fbc77317825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.245025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.244993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0d871ec4bfc496edfc43fbc77317825-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal\" (UID: \"e0d871ec4bfc496edfc43fbc77317825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.257707 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.257691    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:11.345530 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.345502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb470419a847df4b595db3785a098356-config\") pod \"kube-apiserver-proxy-ip-10-0-136-111.ec2.internal\" (UID: \"cb470419a847df4b595db3785a098356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.345530 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.345532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0d871ec4bfc496edfc43fbc77317825-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal\" (UID: \"e0d871ec4bfc496edfc43fbc77317825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.345689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.345550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0d871ec4bfc496edfc43fbc77317825-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal\" (UID: \"e0d871ec4bfc496edfc43fbc77317825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.345689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.345592    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0d871ec4bfc496edfc43fbc77317825-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal\" (UID: \"e0d871ec4bfc496edfc43fbc77317825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.345689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.345601    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/cb470419a847df4b595db3785a098356-config\") pod \"kube-apiserver-proxy-ip-10-0-136-111.ec2.internal\" (UID: \"cb470419a847df4b595db3785a098356\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.345689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.345607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e0d871ec4bfc496edfc43fbc77317825-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal\" (UID: \"e0d871ec4bfc496edfc43fbc77317825\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:11.358789 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.358730    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:11.459576 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:11.459542    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:11.541801 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:11.541765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:12.754066 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:12.754049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:12.766130 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:12.766114    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:12.866813 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:12.866708    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:12.967229 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:12.967193    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:13.067786 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:13.067746    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:13.082943 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.082918    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:10:13.083100 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.083068    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:10:13.166092 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.165999    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:13.168298 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:13.168281    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-111.ec2.internal\" not found"
Jul 28 17:10:13.188681 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.188613    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:05:12 +0000 UTC" deadline="2028-05-02 19:24:14.948554886 +0000 UTC"
Jul 28 17:10:13.188681 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.188680    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15458h14m1.759878171s"
Jul 28 17:10:13.214688 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.214660    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:13.248967 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.248938    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:13.261337 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.261306    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:13.262350 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.262330    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal"
Jul 28 17:10:13.274694 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.274665    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:10:13.326584 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:13.326549    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d871ec4bfc496edfc43fbc77317825.slice/crio-2452fc5e47c4c6a76203b23faf4160416806058122e7f1384666077dd60762f3 WatchSource:0}: Error finding container 2452fc5e47c4c6a76203b23faf4160416806058122e7f1384666077dd60762f3: Status 404 returned error can't find the container with id 2452fc5e47c4c6a76203b23faf4160416806058122e7f1384666077dd60762f3
Jul 28 17:10:13.332304 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.332290    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:10:13.376192 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:13.376162    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:13.394226 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:13.394194    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb470419a847df4b595db3785a098356.slice/crio-d92b44ab0ef4ea7f0c6221342f136270760472941ae3db337af6d2652095e330 WatchSource:0}: Error finding container d92b44ab0ef4ea7f0c6221342f136270760472941ae3db337af6d2652095e330: Status 404 returned error can't find the container with id d92b44ab0ef4ea7f0c6221342f136270760472941ae3db337af6d2652095e330
Jul 28 17:10:14.127257 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.127216    2579 apiserver.go:52] "Watching apiserver"
Jul 28 17:10:14.135973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.135944    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:10:14.137025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.136998    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-f8nqb","kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c","openshift-cluster-node-tuning-operator/tuned-t5xp7","openshift-image-registry/node-ca-4w8n8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal","openshift-multus/multus-additional-cni-plugins-57cvt","openshift-multus/multus-j4jsd","kube-system/global-pull-secret-syncer-x8kpv","openshift-dns/node-resolver-jh65p","openshift-multus/network-metrics-daemon-bx6mn","openshift-network-diagnostics/network-check-target-qr9ct","openshift-network-operator/iptables-alerter-k7jr9","openshift-ovn-kubernetes/ovnkube-node-b92c2"]
Jul 28 17:10:14.138932 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.138911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.140624 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.140602    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.141959 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.141941    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:10:14.142120 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.142096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.142233 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.142185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-skqtd\""
Jul 28 17:10:14.142233 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.142216    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:10:14.142340 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.142259    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:10:14.142485 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.142468    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:10:14.142609 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.142594    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:10:14.143595 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.143575    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:14.143689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.143633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.144763 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.144745    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:14.144934 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.144906    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:10:14.144998 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.144972    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-982pd\""
Jul 28 17:10:14.145235 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.145220    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.147632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.146602    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:10:14.147632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.146628    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9q7vj\""
Jul 28 17:10:14.147632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.146785    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:10:14.147632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.146938    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:10:14.147632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.146991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.147632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.147183    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:10:14.148371 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.147844    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-85wnf\""
Jul 28 17:10:14.148371 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.147971    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:10:14.148371 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.148057    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:10:14.148846 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.148571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hzjbl\""
Jul 28 17:10:14.148846 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.148621    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:10:14.148846 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.148832    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:10:14.149033 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.149003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:14.149119 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.149095    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:14.150127 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.150108    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:10:14.150744 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.150702    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:10:14.152467 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.151964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:14.152467 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.152208    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:10:14.153241 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.153144    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kxzfz\""
Jul 28 17:10:14.153423 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.153333    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:10:14.153497 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.153464    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:10:14.153709 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.153693    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:10:14.154182 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.153793    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:10:14.154304 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.154289    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.155380 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.155330    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:10:14.156480 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.155853    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9sw7f\""
Jul 28 17:10:14.157601 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.157181    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:10:14.157601 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.157203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.158037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.157820    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nvqpg\""
Jul 28 17:10:14.158037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.157859    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:10:14.159319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.158973    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:14.159319 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.159040    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:14.159319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-device-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.159319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxkmj\" (UniqueName: \"kubernetes.io/projected/13523199-d290-434c-ba05-71c05034b7e7-kube-api-access-lxkmj\") pod \"iptables-alerter-k7jr9\" (UID: \"13523199-d290-434c-ba05-71c05034b7e7\") " pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.159319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-registration-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.159319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-run\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.159319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159272    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cdead62f-00ab-4c05-bf46-50fc2edff78d-ovnkube-script-lib\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.159319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-sys-fs\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-systemd\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65z57\" (UniqueName: \"kubernetes.io/projected/4626f0ae-30f4-4c82-9419-f817a63bbdad-kube-api-access-65z57\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13523199-d290-434c-ba05-71c05034b7e7-iptables-alerter-script\") pod \"iptables-alerter-k7jr9\" (UID: \"13523199-d290-434c-ba05-71c05034b7e7\") " pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e925228f-d166-4239-8f86-dad787c3e6ee-serviceca\") pod \"node-ca-4w8n8\" (UID: \"e925228f-d166-4239-8f86-dad787c3e6ee\") " pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76de2b66-aaa2-426f-b965-89bd9d61a207-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-kubernetes\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-host\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-lib-modules\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-log-socket\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cdead62f-00ab-4c05-bf46-50fc2edff78d-ovnkube-config\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159632    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xtl\" (UniqueName: \"kubernetes.io/projected/e925228f-d166-4239-8f86-dad787c3e6ee-kube-api-access-r5xtl\") pod \"node-ca-4w8n8\" (UID: \"e925228f-d166-4239-8f86-dad787c3e6ee\") " pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e9a6f3f-f55c-471c-8678-b604feccff48-agent-certs\") pod \"konnectivity-agent-f8nqb\" (UID: \"3e9a6f3f-f55c-471c-8678-b604feccff48\") " pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159677    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-cni-netd\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.159740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-sysconfig\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-var-lib-openvswitch\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-etc-openvswitch\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-os-release\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-tuned\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-kubelet\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.159969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-run-ovn\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160032    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-systemd-units\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-sysctl-conf\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-cni-bin\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s86b\" (UniqueName: \"kubernetes.io/projected/cdead62f-00ab-4c05-bf46-50fc2edff78d-kube-api-access-6s86b\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-sys\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-run-systemd\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdead62f-00ab-4c05-bf46-50fc2edff78d-ovn-node-metrics-cert\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.160604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160282    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-var-lib-kubelet\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29dafc6b-d587-47f3-9ea9-f10cacb103fe-dbus\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-socket-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-sysctl-d\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160380    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4626f0ae-30f4-4c82-9419-f817a63bbdad-tmp\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-slash\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-run-openvswitch\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cdead62f-00ab-4c05-bf46-50fc2edff78d-env-overrides\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcnlr\" (UniqueName: \"kubernetes.io/projected/a6cdf093-237d-45ff-ac83-e991ebb2e51c-kube-api-access-qcnlr\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13523199-d290-434c-ba05-71c05034b7e7-host-slash\") pod \"iptables-alerter-k7jr9\" (UID: \"13523199-d290-434c-ba05-71c05034b7e7\") " pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-system-cni-dir\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160565    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-cnibin\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76de2b66-aaa2-426f-b965-89bd9d61a207-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-modprobe-d\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.161336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29dafc6b-d587-47f3-9ea9-f10cacb103fe-kubelet-config\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:14.162036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76de2b66-aaa2-426f-b965-89bd9d61a207-cni-binary-copy\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.162036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sll2\" (UniqueName: \"kubernetes.io/projected/76de2b66-aaa2-426f-b965-89bd9d61a207-kube-api-access-8sll2\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.162036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-run-netns\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.162036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-node-log\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.162036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-run-ovn-kubernetes\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.162036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e925228f-d166-4239-8f86-dad787c3e6ee-host\") pod \"node-ca-4w8n8\" (UID: \"e925228f-d166-4239-8f86-dad787c3e6ee\") " pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.162036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.160852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e9a6f3f-f55c-471c-8678-b604feccff48-konnectivity-ca\") pod \"konnectivity-agent-f8nqb\" (UID: \"3e9a6f3f-f55c-471c-8678-b604feccff48\") " pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:14.162036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.161691    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ng6cl\""
Jul 28 17:10:14.162036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.161893    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:10:14.162447 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.162164    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:10:14.163468 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.163449    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:14.163468 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.163508    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:14.165921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.165762    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:10:14.235648 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.235615    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hwbbl"
Jul 28 17:10:14.245448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.245393    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hwbbl"
Jul 28 17:10:14.250417 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.250395    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:10:14.261416 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e9a6f3f-f55c-471c-8678-b604feccff48-agent-certs\") pod \"konnectivity-agent-f8nqb\" (UID: \"3e9a6f3f-f55c-471c-8678-b604feccff48\") " pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:14.261563 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34d3b868-e31e-4fd6-9b00-f117a39ebbc3-hosts-file\") pod \"node-resolver-jh65p\" (UID: \"34d3b868-e31e-4fd6-9b00-f117a39ebbc3\") " pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.261563 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-cni-netd\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.261563 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-conf-dir\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.261563 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-sysconfig\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.261563 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-var-lib-openvswitch\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.261821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-etc-openvswitch\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.261821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-system-cni-dir\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.261821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28c3098d-a6c1-4700-95fe-93e39e621f83-cni-binary-copy\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.261821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-os-release\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.261821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.261821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-tuned\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.261821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-kubelet\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.261821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-run-ovn\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.261821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-etc-kubernetes\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261843    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-systemd-units\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261906    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-sysctl-conf\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-cni-bin\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.261986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s86b\" (UniqueName: \"kubernetes.io/projected/cdead62f-00ab-4c05-bf46-50fc2edff78d-kube-api-access-6s86b\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-etc-openvswitch\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-sys\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-kubelet\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-var-lib-openvswitch\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-run-systemd\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdead62f-00ab-4c05-bf46-50fc2edff78d-ovn-node-metrics-cert\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262187    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sz24\" (UniqueName: \"kubernetes.io/projected/34d3b868-e31e-4fd6-9b00-f117a39ebbc3-kube-api-access-8sz24\") pod \"node-resolver-jh65p\" (UID: \"34d3b868-e31e-4fd6-9b00-f117a39ebbc3\") " pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-var-lib-cni-bin\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-etc-selinux\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.262266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-var-lib-kubelet\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-cni-bin\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262341    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29dafc6b-d587-47f3-9ea9-f10cacb103fe-dbus\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-systemd-units\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-os-release\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262424    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-cni-netd\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-run-ovn\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-run-netns\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-var-lib-kubelet\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-sysctl-conf\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-socket-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.262953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.262899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29dafc6b-d587-47f3-9ea9-f10cacb103fe-dbus\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-os-release\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-sysctl-d\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-socket-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4626f0ae-30f4-4c82-9419-f817a63bbdad-tmp\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-slash\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-run-openvswitch\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-sysctl-d\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cdead62f-00ab-4c05-bf46-50fc2edff78d-env-overrides\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-run-systemd\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-sys\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-slash\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263350    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-run-openvswitch\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263330    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljwcr\" (UniqueName: \"kubernetes.io/projected/19846f2a-05eb-485f-8ae6-67e7090bc346-kube-api-access-ljwcr\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-socket-dir-parent\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcnlr\" (UniqueName: \"kubernetes.io/projected/a6cdf093-237d-45ff-ac83-e991ebb2e51c-kube-api-access-qcnlr\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d3b868-e31e-4fd6-9b00-f117a39ebbc3-tmp-dir\") pod \"node-resolver-jh65p\" (UID: \"34d3b868-e31e-4fd6-9b00-f117a39ebbc3\") " pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.263626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-hostroot\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13523199-d290-434c-ba05-71c05034b7e7-host-slash\") pod \"iptables-alerter-k7jr9\" (UID: \"13523199-d290-434c-ba05-71c05034b7e7\") " pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-sysconfig\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-system-cni-dir\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263624    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/13523199-d290-434c-ba05-71c05034b7e7-host-slash\") pod \"iptables-alerter-k7jr9\" (UID: \"13523199-d290-434c-ba05-71c05034b7e7\") " pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-cnibin\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76de2b66-aaa2-426f-b965-89bd9d61a207-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-cnibin\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/76de2b66-aaa2-426f-b965-89bd9d61a207-system-cni-dir\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-modprobe-d\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cdead62f-00ab-4c05-bf46-50fc2edff78d-env-overrides\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29dafc6b-d587-47f3-9ea9-f10cacb103fe-kubelet-config\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-modprobe-d\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-var-lib-cni-multus\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76de2b66-aaa2-426f-b965-89bd9d61a207-cni-binary-copy\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.264334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.263999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sll2\" (UniqueName: \"kubernetes.io/projected/76de2b66-aaa2-426f-b965-89bd9d61a207-kube-api-access-8sll2\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29dafc6b-d587-47f3-9ea9-f10cacb103fe-kubelet-config\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-run-netns\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-run-netns\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-node-log\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-run-ovn-kubernetes\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e925228f-d166-4239-8f86-dad787c3e6ee-host\") pod \"node-ca-4w8n8\" (UID: \"e925228f-d166-4239-8f86-dad787c3e6ee\") " pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264193    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e9a6f3f-f55c-471c-8678-b604feccff48-konnectivity-ca\") pod \"konnectivity-agent-f8nqb\" (UID: \"3e9a6f3f-f55c-471c-8678-b604feccff48\") " pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-device-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxkmj\" (UniqueName: \"kubernetes.io/projected/13523199-d290-434c-ba05-71c05034b7e7-kube-api-access-lxkmj\") pod \"iptables-alerter-k7jr9\" (UID: \"13523199-d290-434c-ba05-71c05034b7e7\") " pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-run-multus-certs\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-registration-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/76de2b66-aaa2-426f-b965-89bd9d61a207-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-run\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cdead62f-00ab-4c05-bf46-50fc2edff78d-ovnkube-script-lib\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-host-run-ovn-kubernetes\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.265162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-cni-dir\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-run-k8s-cni-cncf-io\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-node-log\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-daemon-config\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/76de2b66-aaa2-426f-b965-89bd9d61a207-cni-binary-copy\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-registration-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-sys-fs\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-sys-fs\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-systemd\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.264646    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65z57\" (UniqueName: \"kubernetes.io/projected/4626f0ae-30f4-4c82-9419-f817a63bbdad-kube-api-access-65z57\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13523199-d290-434c-ba05-71c05034b7e7-iptables-alerter-script\") pod \"iptables-alerter-k7jr9\" (UID: \"13523199-d290-434c-ba05-71c05034b7e7\") " pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.264712    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret podName:29dafc6b-d587-47f3-9ea9-f10cacb103fe nodeName:}" failed. No retries permitted until 2026-07-28 17:10:14.764691168 +0000 UTC m=+2.993890255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret") pod "global-pull-secret-syncer-x8kpv" (UID: "29dafc6b-d587-47f3-9ea9-f10cacb103fe") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264736    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a6cdf093-237d-45ff-ac83-e991ebb2e51c-device-dir\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e925228f-d166-4239-8f86-dad787c3e6ee-serviceca\") pod \"node-ca-4w8n8\" (UID: \"e925228f-d166-4239-8f86-dad787c3e6ee\") " pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfrj8\" (UniqueName: \"kubernetes.io/projected/28c3098d-a6c1-4700-95fe-93e39e621f83-kube-api-access-kfrj8\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264830    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3e9a6f3f-f55c-471c-8678-b604feccff48-konnectivity-ca\") pod \"konnectivity-agent-f8nqb\" (UID: \"3e9a6f3f-f55c-471c-8678-b604feccff48\") " pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76de2b66-aaa2-426f-b965-89bd9d61a207-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-kubernetes\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-host\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-cnibin\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-var-lib-kubelet\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh8np\" (UniqueName: \"kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np\") pod \"network-check-target-qr9ct\" (UID: \"0569b22a-fdc0-4877-8d81-6b2f9debea9c\") " pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.264999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-lib-modules\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-log-socket\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-host\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cdead62f-00ab-4c05-bf46-50fc2edff78d-ovnkube-config\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xtl\" (UniqueName: \"kubernetes.io/projected/e925228f-d166-4239-8f86-dad787c3e6ee-kube-api-access-r5xtl\") pod \"node-ca-4w8n8\" (UID: \"e925228f-d166-4239-8f86-dad787c3e6ee\") " pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e925228f-d166-4239-8f86-dad787c3e6ee-host\") pod \"node-ca-4w8n8\" (UID: \"e925228f-d166-4239-8f86-dad787c3e6ee\") " pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-systemd\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/76de2b66-aaa2-426f-b965-89bd9d61a207-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265509    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-lib-modules\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-kubernetes\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.266728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cdead62f-00ab-4c05-bf46-50fc2edff78d-log-socket\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.267454 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e925228f-d166-4239-8f86-dad787c3e6ee-serviceca\") pod \"node-ca-4w8n8\" (UID: \"e925228f-d166-4239-8f86-dad787c3e6ee\") " pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.267454 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/13523199-d290-434c-ba05-71c05034b7e7-iptables-alerter-script\") pod \"iptables-alerter-k7jr9\" (UID: \"13523199-d290-434c-ba05-71c05034b7e7\") " pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.267454 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4626f0ae-30f4-4c82-9419-f817a63bbdad-run\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.267454 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.265998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cdead62f-00ab-4c05-bf46-50fc2edff78d-ovnkube-script-lib\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.267454 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.266423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cdead62f-00ab-4c05-bf46-50fc2edff78d-ovnkube-config\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.268295 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.267567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3e9a6f3f-f55c-471c-8678-b604feccff48-agent-certs\") pod \"konnectivity-agent-f8nqb\" (UID: \"3e9a6f3f-f55c-471c-8678-b604feccff48\") " pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:14.268295 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.267852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4626f0ae-30f4-4c82-9419-f817a63bbdad-etc-tuned\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.270016 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.269986    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4626f0ae-30f4-4c82-9419-f817a63bbdad-tmp\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.270836 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.270812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdead62f-00ab-4c05-bf46-50fc2edff78d-ovn-node-metrics-cert\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.273023 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.272997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sll2\" (UniqueName: \"kubernetes.io/projected/76de2b66-aaa2-426f-b965-89bd9d61a207-kube-api-access-8sll2\") pod \"multus-additional-cni-plugins-57cvt\" (UID: \"76de2b66-aaa2-426f-b965-89bd9d61a207\") " pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.274023 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.274001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcnlr\" (UniqueName: \"kubernetes.io/projected/a6cdf093-237d-45ff-ac83-e991ebb2e51c-kube-api-access-qcnlr\") pod \"aws-ebs-csi-driver-node-hkk4c\" (UID: \"a6cdf093-237d-45ff-ac83-e991ebb2e51c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.275465 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.275397    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s86b\" (UniqueName: \"kubernetes.io/projected/cdead62f-00ab-4c05-bf46-50fc2edff78d-kube-api-access-6s86b\") pod \"ovnkube-node-b92c2\" (UID: \"cdead62f-00ab-4c05-bf46-50fc2edff78d\") " pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.275988 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.275938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxkmj\" (UniqueName: \"kubernetes.io/projected/13523199-d290-434c-ba05-71c05034b7e7-kube-api-access-lxkmj\") pod \"iptables-alerter-k7jr9\" (UID: \"13523199-d290-434c-ba05-71c05034b7e7\") " pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.277052 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.277029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65z57\" (UniqueName: \"kubernetes.io/projected/4626f0ae-30f4-4c82-9419-f817a63bbdad-kube-api-access-65z57\") pod \"tuned-t5xp7\" (UID: \"4626f0ae-30f4-4c82-9419-f817a63bbdad\") " pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.277943 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.277489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xtl\" (UniqueName: \"kubernetes.io/projected/e925228f-d166-4239-8f86-dad787c3e6ee-kube-api-access-r5xtl\") pod \"node-ca-4w8n8\" (UID: \"e925228f-d166-4239-8f86-dad787c3e6ee\") " pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.320500 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.320443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal" event={"ID":"e0d871ec4bfc496edfc43fbc77317825","Type":"ContainerStarted","Data":"2452fc5e47c4c6a76203b23faf4160416806058122e7f1384666077dd60762f3"}
Jul 28 17:10:14.321998 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.321969    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal" event={"ID":"cb470419a847df4b595db3785a098356","Type":"ContainerStarted","Data":"d92b44ab0ef4ea7f0c6221342f136270760472941ae3db337af6d2652095e330"}
Jul 28 17:10:14.366327 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-run-k8s-cni-cncf-io\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366327 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-daemon-config\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:14.366566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfrj8\" (UniqueName: \"kubernetes.io/projected/28c3098d-a6c1-4700-95fe-93e39e621f83-kube-api-access-kfrj8\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-run-k8s-cni-cncf-io\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366456    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-cnibin\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-var-lib-kubelet\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8np\" (UniqueName: \"kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np\") pod \"network-check-target-qr9ct\" (UID: \"0569b22a-fdc0-4877-8d81-6b2f9debea9c\") " pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:14.366566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34d3b868-e31e-4fd6-9b00-f117a39ebbc3-hosts-file\") pod \"node-resolver-jh65p\" (UID: \"34d3b868-e31e-4fd6-9b00-f117a39ebbc3\") " pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.366566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-conf-dir\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-system-cni-dir\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.366595    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28c3098d-a6c1-4700-95fe-93e39e621f83-cni-binary-copy\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.366728    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs podName:19846f2a-05eb-485f-8ae6-67e7090bc346 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:14.866637795 +0000 UTC m=+3.095836870 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs") pod "network-metrics-daemon-bx6mn" (UID: "19846f2a-05eb-485f-8ae6-67e7090bc346") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-etc-kubernetes\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-cnibin\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sz24\" (UniqueName: \"kubernetes.io/projected/34d3b868-e31e-4fd6-9b00-f117a39ebbc3-kube-api-access-8sz24\") pod \"node-resolver-jh65p\" (UID: \"34d3b868-e31e-4fd6-9b00-f117a39ebbc3\") " pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-conf-dir\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-var-lib-cni-bin\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/34d3b868-e31e-4fd6-9b00-f117a39ebbc3-hosts-file\") pod \"node-resolver-jh65p\" (UID: \"34d3b868-e31e-4fd6-9b00-f117a39ebbc3\") " pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-os-release\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-os-release\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-run-netns\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.366944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljwcr\" (UniqueName: \"kubernetes.io/projected/19846f2a-05eb-485f-8ae6-67e7090bc346-kube-api-access-ljwcr\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-socket-dir-parent\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d3b868-e31e-4fd6-9b00-f117a39ebbc3-tmp-dir\") pod \"node-resolver-jh65p\" (UID: \"34d3b868-e31e-4fd6-9b00-f117a39ebbc3\") " pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-hostroot\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-var-lib-cni-multus\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367014    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-daemon-config\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-run-multus-certs\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.366814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-var-lib-kubelet\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/28c3098d-a6c1-4700-95fe-93e39e621f83-cni-binary-copy\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-etc-kubernetes\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-system-cni-dir\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-run-netns\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-cni-dir\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-var-lib-cni-multus\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367197    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-cni-dir\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-var-lib-cni-bin\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367211    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-hostroot\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-host-run-multus-certs\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.367691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34d3b868-e31e-4fd6-9b00-f117a39ebbc3-tmp-dir\") pod \"node-resolver-jh65p\" (UID: \"34d3b868-e31e-4fd6-9b00-f117a39ebbc3\") " pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.368468 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.367385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/28c3098d-a6c1-4700-95fe-93e39e621f83-multus-socket-dir-parent\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.378033 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.377961    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:14.378033 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.377989    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:14.378033 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.378003    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lh8np for pod openshift-network-diagnostics/network-check-target-qr9ct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:14.378270 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.378069    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np podName:0569b22a-fdc0-4877-8d81-6b2f9debea9c nodeName:}" failed. No retries permitted until 2026-07-28 17:10:14.878049275 +0000 UTC m=+3.107248370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8np" (UniqueName: "kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np") pod "network-check-target-qr9ct" (UID: "0569b22a-fdc0-4877-8d81-6b2f9debea9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:14.378369 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.378348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfrj8\" (UniqueName: \"kubernetes.io/projected/28c3098d-a6c1-4700-95fe-93e39e621f83-kube-api-access-kfrj8\") pod \"multus-j4jsd\" (UID: \"28c3098d-a6c1-4700-95fe-93e39e621f83\") " pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.382431 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.382407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sz24\" (UniqueName: \"kubernetes.io/projected/34d3b868-e31e-4fd6-9b00-f117a39ebbc3-kube-api-access-8sz24\") pod \"node-resolver-jh65p\" (UID: \"34d3b868-e31e-4fd6-9b00-f117a39ebbc3\") " pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.384040 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.384018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljwcr\" (UniqueName: \"kubernetes.io/projected/19846f2a-05eb-485f-8ae6-67e7090bc346-kube-api-access-ljwcr\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:14.451669 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.451634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-57cvt"
Jul 28 17:10:14.461570 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.461544    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k7jr9"
Jul 28 17:10:14.471979 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.471951    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c"
Jul 28 17:10:14.478618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.478593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t5xp7"
Jul 28 17:10:14.486539 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.486518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4w8n8"
Jul 28 17:10:14.491424 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.491399    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:10:14.494847 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.494825    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:14.502852 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.502822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:14.509922 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.509899    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j4jsd"
Jul 28 17:10:14.517064 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.517041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jh65p"
Jul 28 17:10:14.523041 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:14.523014    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13523199_d290_434c_ba05_71c05034b7e7.slice/crio-63b14fb83f0b7689cf1295505a045ad8f48ea0d774917fd325de165bf14bccd7 WatchSource:0}: Error finding container 63b14fb83f0b7689cf1295505a045ad8f48ea0d774917fd325de165bf14bccd7: Status 404 returned error can't find the container with id 63b14fb83f0b7689cf1295505a045ad8f48ea0d774917fd325de165bf14bccd7
Jul 28 17:10:14.528280 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:14.528253    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdead62f_00ab_4c05_bf46_50fc2edff78d.slice/crio-a0a9b2818410c14e6e976f69d2f4fa3e9176ecd30f4713e54fe69e688f570936 WatchSource:0}: Error finding container a0a9b2818410c14e6e976f69d2f4fa3e9176ecd30f4713e54fe69e688f570936: Status 404 returned error can't find the container with id a0a9b2818410c14e6e976f69d2f4fa3e9176ecd30f4713e54fe69e688f570936
Jul 28 17:10:14.533819 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:14.533793    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode925228f_d166_4239_8f86_dad787c3e6ee.slice/crio-84e0d9bbc356d8cf14f0c65bba0f9b8c74afe68ee3d963f7fd9d1a65e760fcd9 WatchSource:0}: Error finding container 84e0d9bbc356d8cf14f0c65bba0f9b8c74afe68ee3d963f7fd9d1a65e760fcd9: Status 404 returned error can't find the container with id 84e0d9bbc356d8cf14f0c65bba0f9b8c74afe68ee3d963f7fd9d1a65e760fcd9
Jul 28 17:10:14.534759 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:14.534709    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9a6f3f_f55c_471c_8678_b604feccff48.slice/crio-547eff32b6be3cb206a94d8a3620329d1269deacd0ca6c7ed02036c5503c0bb3 WatchSource:0}: Error finding container 547eff32b6be3cb206a94d8a3620329d1269deacd0ca6c7ed02036c5503c0bb3: Status 404 returned error can't find the container with id 547eff32b6be3cb206a94d8a3620329d1269deacd0ca6c7ed02036c5503c0bb3
Jul 28 17:10:14.536193 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:14.536164    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76de2b66_aaa2_426f_b965_89bd9d61a207.slice/crio-5f5ecf2f9dddc0ccf63cfd1693e0269c798da1f302b99b60496a85597e467688 WatchSource:0}: Error finding container 5f5ecf2f9dddc0ccf63cfd1693e0269c798da1f302b99b60496a85597e467688: Status 404 returned error can't find the container with id 5f5ecf2f9dddc0ccf63cfd1693e0269c798da1f302b99b60496a85597e467688
Jul 28 17:10:14.537936 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:14.537255    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cdf093_237d_45ff_ac83_e991ebb2e51c.slice/crio-3449a3a6be1a5406083247b324373ea5cfcff5c8900df07350ba2d3fbbd386da WatchSource:0}: Error finding container 3449a3a6be1a5406083247b324373ea5cfcff5c8900df07350ba2d3fbbd386da: Status 404 returned error can't find the container with id 3449a3a6be1a5406083247b324373ea5cfcff5c8900df07350ba2d3fbbd386da
Jul 28 17:10:14.538282 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:14.538259    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4626f0ae_30f4_4c82_9419_f817a63bbdad.slice/crio-0180e0289173e249a3969a0d301098b2ce889faca1aa0cefbdd247ce40bbeb23 WatchSource:0}: Error finding container 0180e0289173e249a3969a0d301098b2ce889faca1aa0cefbdd247ce40bbeb23: Status 404 returned error can't find the container with id 0180e0289173e249a3969a0d301098b2ce889faca1aa0cefbdd247ce40bbeb23
Jul 28 17:10:14.541181 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:14.541153    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c3098d_a6c1_4700_95fe_93e39e621f83.slice/crio-c4826ce4c81f1dc6672835e81e0bd8dce4ac6d4fe9096d17fecaecd8a050889c WatchSource:0}: Error finding container c4826ce4c81f1dc6672835e81e0bd8dce4ac6d4fe9096d17fecaecd8a050889c: Status 404 returned error can't find the container with id c4826ce4c81f1dc6672835e81e0bd8dce4ac6d4fe9096d17fecaecd8a050889c
Jul 28 17:10:14.542694 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:14.542644    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d3b868_e31e_4fd6_9b00_f117a39ebbc3.slice/crio-570da49472d505c9a91a75265ae6c9e969f11888aa43254da869e641237bae2e WatchSource:0}: Error finding container 570da49472d505c9a91a75265ae6c9e969f11888aa43254da869e641237bae2e: Status 404 returned error can't find the container with id 570da49472d505c9a91a75265ae6c9e969f11888aa43254da869e641237bae2e
Jul 28 17:10:14.771471 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.771440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:14.771647 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.771627    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:14.771762 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.771697    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret podName:29dafc6b-d587-47f3-9ea9-f10cacb103fe nodeName:}" failed. No retries permitted until 2026-07-28 17:10:15.771678165 +0000 UTC m=+4.000877238 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret") pod "global-pull-secret-syncer-x8kpv" (UID: "29dafc6b-d587-47f3-9ea9-f10cacb103fe") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:14.872368 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.872336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:14.872532 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.872514    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:14.872604 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.872588    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs podName:19846f2a-05eb-485f-8ae6-67e7090bc346 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:15.872568631 +0000 UTC m=+4.101767704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs") pod "network-metrics-daemon-bx6mn" (UID: "19846f2a-05eb-485f-8ae6-67e7090bc346") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:14.973575 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:14.973402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8np\" (UniqueName: \"kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np\") pod \"network-check-target-qr9ct\" (UID: \"0569b22a-fdc0-4877-8d81-6b2f9debea9c\") " pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:14.973721 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.973592    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:14.973721 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.973617    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:14.973721 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.973631    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lh8np for pod openshift-network-diagnostics/network-check-target-qr9ct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:14.973721 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:14.973698    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np podName:0569b22a-fdc0-4877-8d81-6b2f9debea9c nodeName:}" failed. No retries permitted until 2026-07-28 17:10:15.973677954 +0000 UTC m=+4.202877042 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8np" (UniqueName: "kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np") pod "network-check-target-qr9ct" (UID: "0569b22a-fdc0-4877-8d81-6b2f9debea9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:15.247865 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.247829    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:14 +0000 UTC" deadline="2028-01-24 14:52:38.114125348 +0000 UTC"
Jul 28 17:10:15.247865 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.247865    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13077h42m22.866264435s"
Jul 28 17:10:15.314254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.314235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:15.314348 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.314235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:15.314348 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.314333    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:15.314431 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.314235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:15.314431 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.314419    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:15.314526 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.314503    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:15.334561 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.334394    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0d871ec4bfc496edfc43fbc77317825" containerID="9fff02fe591f2da00361d55403ef59f604a23b6b272b7749b0e3e2d529a19263" exitCode=0
Jul 28 17:10:15.334717 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.334620    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal" event={"ID":"e0d871ec4bfc496edfc43fbc77317825","Type":"ContainerDied","Data":"9fff02fe591f2da00361d55403ef59f604a23b6b272b7749b0e3e2d529a19263"}
Jul 28 17:10:15.343936 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.343215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t5xp7" event={"ID":"4626f0ae-30f4-4c82-9419-f817a63bbdad","Type":"ContainerStarted","Data":"0180e0289173e249a3969a0d301098b2ce889faca1aa0cefbdd247ce40bbeb23"}
Jul 28 17:10:15.347985 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.347905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c" event={"ID":"a6cdf093-237d-45ff-ac83-e991ebb2e51c","Type":"ContainerStarted","Data":"3449a3a6be1a5406083247b324373ea5cfcff5c8900df07350ba2d3fbbd386da"}
Jul 28 17:10:15.355829 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.355803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57cvt" event={"ID":"76de2b66-aaa2-426f-b965-89bd9d61a207","Type":"ContainerStarted","Data":"5f5ecf2f9dddc0ccf63cfd1693e0269c798da1f302b99b60496a85597e467688"}
Jul 28 17:10:15.357437 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.357380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jh65p" event={"ID":"34d3b868-e31e-4fd6-9b00-f117a39ebbc3","Type":"ContainerStarted","Data":"570da49472d505c9a91a75265ae6c9e969f11888aa43254da869e641237bae2e"}
Jul 28 17:10:15.361996 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.361939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j4jsd" event={"ID":"28c3098d-a6c1-4700-95fe-93e39e621f83","Type":"ContainerStarted","Data":"c4826ce4c81f1dc6672835e81e0bd8dce4ac6d4fe9096d17fecaecd8a050889c"}
Jul 28 17:10:15.366102 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.366060    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4w8n8" event={"ID":"e925228f-d166-4239-8f86-dad787c3e6ee","Type":"ContainerStarted","Data":"84e0d9bbc356d8cf14f0c65bba0f9b8c74afe68ee3d963f7fd9d1a65e760fcd9"}
Jul 28 17:10:15.376952 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.376899    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f8nqb" event={"ID":"3e9a6f3f-f55c-471c-8678-b604feccff48","Type":"ContainerStarted","Data":"547eff32b6be3cb206a94d8a3620329d1269deacd0ca6c7ed02036c5503c0bb3"}
Jul 28 17:10:15.382404 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.382300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2" event={"ID":"cdead62f-00ab-4c05-bf46-50fc2edff78d","Type":"ContainerStarted","Data":"a0a9b2818410c14e6e976f69d2f4fa3e9176ecd30f4713e54fe69e688f570936"}
Jul 28 17:10:15.409500 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.409212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k7jr9" event={"ID":"13523199-d290-434c-ba05-71c05034b7e7","Type":"ContainerStarted","Data":"63b14fb83f0b7689cf1295505a045ad8f48ea0d774917fd325de165bf14bccd7"}
Jul 28 17:10:15.782790 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.782028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:15.782790 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.782257    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:15.782790 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.782320    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret podName:29dafc6b-d587-47f3-9ea9-f10cacb103fe nodeName:}" failed. No retries permitted until 2026-07-28 17:10:17.782302833 +0000 UTC m=+6.011501909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret") pod "global-pull-secret-syncer-x8kpv" (UID: "29dafc6b-d587-47f3-9ea9-f10cacb103fe") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:15.883400 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.882806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:15.883400 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.882972    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:15.883400 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.883035    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs podName:19846f2a-05eb-485f-8ae6-67e7090bc346 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:17.883016591 +0000 UTC m=+6.112215667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs") pod "network-metrics-daemon-bx6mn" (UID: "19846f2a-05eb-485f-8ae6-67e7090bc346") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:15.983932 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:15.983886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8np\" (UniqueName: \"kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np\") pod \"network-check-target-qr9ct\" (UID: \"0569b22a-fdc0-4877-8d81-6b2f9debea9c\") " pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:15.984111 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.984099    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:15.984183 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.984119    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:15.984183 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.984131    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lh8np for pod openshift-network-diagnostics/network-check-target-qr9ct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:15.984283 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:15.984189    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np podName:0569b22a-fdc0-4877-8d81-6b2f9debea9c nodeName:}" failed. No retries permitted until 2026-07-28 17:10:17.984170713 +0000 UTC m=+6.213369791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8np" (UniqueName: "kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np") pod "network-check-target-qr9ct" (UID: "0569b22a-fdc0-4877-8d81-6b2f9debea9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:16.248344 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:16.248245    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:05:14 +0000 UTC" deadline="2028-04-29 12:32:13.373265988 +0000 UTC"
Jul 28 17:10:16.248344 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:16.248312    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15379h21m57.124958429s"
Jul 28 17:10:16.422405 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:16.421602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal" event={"ID":"e0d871ec4bfc496edfc43fbc77317825","Type":"ContainerStarted","Data":"7c6131e61459b7a34ec2357194f180a5c68d62729ed42fa830be03e64ea9d128"}
Jul 28 17:10:16.434640 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:16.434152    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal" event={"ID":"cb470419a847df4b595db3785a098356","Type":"ContainerStarted","Data":"3f179a38a228807e5ae27d08d372562b7ed1d553f88771df48f16c1d6f18c0e0"}
Jul 28 17:10:16.441834 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:16.441775    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-111.ec2.internal" podStartSLOduration=3.441757448 podStartE2EDuration="3.441757448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:16.441229929 +0000 UTC m=+4.670429028" watchObservedRunningTime="2026-07-28 17:10:16.441757448 +0000 UTC m=+4.670956551"
Jul 28 17:10:17.313914 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:17.313835    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:17.314395 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:17.313976    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:17.314476 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:17.314392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:17.314529 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:17.314490    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:17.314601 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:17.314554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:17.314651 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:17.314625    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:17.800303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:17.800222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:17.800456 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:17.800354    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:17.800456 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:17.800419    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret podName:29dafc6b-d587-47f3-9ea9-f10cacb103fe nodeName:}" failed. No retries permitted until 2026-07-28 17:10:21.800402835 +0000 UTC m=+10.029601913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret") pod "global-pull-secret-syncer-x8kpv" (UID: "29dafc6b-d587-47f3-9ea9-f10cacb103fe") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:17.900987 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:17.900948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:17.901193 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:17.901134    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:17.901263 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:17.901195    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs podName:19846f2a-05eb-485f-8ae6-67e7090bc346 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:21.901177153 +0000 UTC m=+10.130376230 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs") pod "network-metrics-daemon-bx6mn" (UID: "19846f2a-05eb-485f-8ae6-67e7090bc346") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:18.002072 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:18.002034    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8np\" (UniqueName: \"kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np\") pod \"network-check-target-qr9ct\" (UID: \"0569b22a-fdc0-4877-8d81-6b2f9debea9c\") " pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:18.002250 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:18.002219    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:18.002250 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:18.002240    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:18.002250 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:18.002252    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lh8np for pod openshift-network-diagnostics/network-check-target-qr9ct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:18.002405 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:18.002308    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np podName:0569b22a-fdc0-4877-8d81-6b2f9debea9c nodeName:}" failed. No retries permitted until 2026-07-28 17:10:22.002290726 +0000 UTC m=+10.231489816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8np" (UniqueName: "kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np") pod "network-check-target-qr9ct" (UID: "0569b22a-fdc0-4877-8d81-6b2f9debea9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:19.313991 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:19.313956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:19.314502 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:19.313968    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:19.314502 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:19.314063    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:19.314502 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:19.314126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:19.314502 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:19.314266    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:19.314502 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:19.314357    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:21.314221 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:21.314144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:21.314655 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:21.314260    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:21.314655 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:21.314630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:21.314766 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:21.314734    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:21.314818 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:21.314782    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:21.314873 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:21.314852    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:21.835409 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:21.835372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:21.835582 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:21.835563    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:21.835646 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:21.835633    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret podName:29dafc6b-d587-47f3-9ea9-f10cacb103fe nodeName:}" failed. No retries permitted until 2026-07-28 17:10:29.835612934 +0000 UTC m=+18.064812013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret") pod "global-pull-secret-syncer-x8kpv" (UID: "29dafc6b-d587-47f3-9ea9-f10cacb103fe") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:21.936460 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:21.936425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:21.936633 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:21.936586    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:21.936686 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:21.936661    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs podName:19846f2a-05eb-485f-8ae6-67e7090bc346 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:29.936640095 +0000 UTC m=+18.165839186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs") pod "network-metrics-daemon-bx6mn" (UID: "19846f2a-05eb-485f-8ae6-67e7090bc346") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:22.037091 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:22.037004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8np\" (UniqueName: \"kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np\") pod \"network-check-target-qr9ct\" (UID: \"0569b22a-fdc0-4877-8d81-6b2f9debea9c\") " pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:22.037269 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:22.037194    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:22.037269 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:22.037218    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:22.037269 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:22.037230    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lh8np for pod openshift-network-diagnostics/network-check-target-qr9ct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:22.037413 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:22.037289    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np podName:0569b22a-fdc0-4877-8d81-6b2f9debea9c nodeName:}" failed. No retries permitted until 2026-07-28 17:10:30.037270454 +0000 UTC m=+18.266469533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8np" (UniqueName: "kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np") pod "network-check-target-qr9ct" (UID: "0569b22a-fdc0-4877-8d81-6b2f9debea9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:23.313632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:23.313545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:23.313632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:23.313573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:23.314171 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:23.313669    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:23.314171 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:23.313780    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:23.314171 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:23.313853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:23.314171 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:23.313928    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:25.313536 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:25.313502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:25.313536 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:25.313526    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:25.313536 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:25.313535    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:25.314036 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:25.313629    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:25.314036 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:25.313735    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:25.314036 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:25.313802    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:27.313632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:27.313591    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:27.314150 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:27.313702    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:27.314150 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:27.313610    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:27.314150 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:27.313592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:27.314150 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:27.313797    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:27.314150 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:27.313893    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:29.314390 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:29.314356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:29.314789 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:29.314356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:29.314789 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:29.314473    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:29.314789 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:29.314366    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:29.314789 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:29.314585    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:29.314789 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:29.314651    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:29.895969 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:29.895932    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:29.896146 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:29.896101    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:29.896222 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:29.896178    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret podName:29dafc6b-d587-47f3-9ea9-f10cacb103fe nodeName:}" failed. No retries permitted until 2026-07-28 17:10:45.896161288 +0000 UTC m=+34.125360363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret") pod "global-pull-secret-syncer-x8kpv" (UID: "29dafc6b-d587-47f3-9ea9-f10cacb103fe") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:29.996665 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:29.996630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:29.996835 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:29.996780    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:29.996950 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:29.996840    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs podName:19846f2a-05eb-485f-8ae6-67e7090bc346 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:45.996822639 +0000 UTC m=+34.226021712 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs") pod "network-metrics-daemon-bx6mn" (UID: "19846f2a-05eb-485f-8ae6-67e7090bc346") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:30.097150 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:30.097111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8np\" (UniqueName: \"kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np\") pod \"network-check-target-qr9ct\" (UID: \"0569b22a-fdc0-4877-8d81-6b2f9debea9c\") " pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:30.097334 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:30.097313    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:30.097405 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:30.097341    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:30.097405 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:30.097356    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lh8np for pod openshift-network-diagnostics/network-check-target-qr9ct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:30.097493 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:30.097424    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np podName:0569b22a-fdc0-4877-8d81-6b2f9debea9c nodeName:}" failed. No retries permitted until 2026-07-28 17:10:46.097403555 +0000 UTC m=+34.326602648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8np" (UniqueName: "kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np") pod "network-check-target-qr9ct" (UID: "0569b22a-fdc0-4877-8d81-6b2f9debea9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:31.314039 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:31.313992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:31.314039 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:31.314022    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:31.314039 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:31.314041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:31.314556 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:31.314166    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:31.314556 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:31.314314    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:31.314556 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:31.314397    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:33.313658 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:33.313620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:33.314068 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:33.313664    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:33.314068 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:33.313707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:33.314068 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:33.313716    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:33.314068 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:33.313769    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:33.314068 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:33.313876    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:34.473516 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.473032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jh65p" event={"ID":"34d3b868-e31e-4fd6-9b00-f117a39ebbc3","Type":"ContainerStarted","Data":"9f68ffb578c009e76b32fda5b2e180c3e2b664689747b5bac205b09809251a16"}
Jul 28 17:10:34.475160 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.475126    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j4jsd" event={"ID":"28c3098d-a6c1-4700-95fe-93e39e621f83","Type":"ContainerStarted","Data":"43274c3beb57e3874f57999b03dfaef886720407d06bd529427c23583c429d7f"}
Jul 28 17:10:34.476673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.476647    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4w8n8" event={"ID":"e925228f-d166-4239-8f86-dad787c3e6ee","Type":"ContainerStarted","Data":"73ff50c42a9473fa4ddadf900c6cd9591cfc19da0f8212d4c6fcd5068b73b9cc"}
Jul 28 17:10:34.478477 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.478453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f8nqb" event={"ID":"3e9a6f3f-f55c-471c-8678-b604feccff48","Type":"ContainerStarted","Data":"9e05a82d0e851b67613b5bda33d2dc6c5cee6f3f8470345a4de354598ad5c4a4"}
Jul 28 17:10:34.481297 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.481271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2" event={"ID":"cdead62f-00ab-4c05-bf46-50fc2edff78d","Type":"ContainerStarted","Data":"1d4cccf2b2825b0a3521a0d20995c7b2de8bc9594fc3c67eee42ec4bfa229dce"}
Jul 28 17:10:34.481451 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.481433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2" event={"ID":"cdead62f-00ab-4c05-bf46-50fc2edff78d","Type":"ContainerStarted","Data":"93bc6a09c62b10be93ad33ae1c9e9f72316353af7fc2342353471a6dbeb58fdf"}
Jul 28 17:10:34.481532 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.481494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2" event={"ID":"cdead62f-00ab-4c05-bf46-50fc2edff78d","Type":"ContainerStarted","Data":"7c43d9e962976d3b18be013bdb38b9e5cef4059e90a2de34c672bd102979f02b"}
Jul 28 17:10:34.481588 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.481542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2" event={"ID":"cdead62f-00ab-4c05-bf46-50fc2edff78d","Type":"ContainerStarted","Data":"c25c9264f477b36225a26ac63efb8d275bfef4dbe8e01ba945da4b4cfbe65bcb"}
Jul 28 17:10:34.481643 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.481588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2" event={"ID":"cdead62f-00ab-4c05-bf46-50fc2edff78d","Type":"ContainerStarted","Data":"d5742e2e20ad04a0af3f0b4ba372bd11ddd689c8bf76ab4343cd35d8845653e2"}
Jul 28 17:10:34.482723 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.482701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t5xp7" event={"ID":"4626f0ae-30f4-4c82-9419-f817a63bbdad","Type":"ContainerStarted","Data":"6678bba7e97912d603838bbdbc24512a7236168cbccfb235be2a71945d13afcb"}
Jul 28 17:10:34.484338 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.484319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c" event={"ID":"a6cdf093-237d-45ff-ac83-e991ebb2e51c","Type":"ContainerStarted","Data":"ff7a250129b7bcd5a13f1376cd2395cd3998c0d2d5cb0690a707e72a41cae565"}
Jul 28 17:10:34.485987 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.485964    2579 generic.go:356] "Generic (PLEG): container finished" podID="76de2b66-aaa2-426f-b965-89bd9d61a207" containerID="53da590249904abef4701585f144fea80b4409b5d8bafaf36ade598da4676c91" exitCode=0
Jul 28 17:10:34.486106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.485998    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57cvt" event={"ID":"76de2b66-aaa2-426f-b965-89bd9d61a207","Type":"ContainerDied","Data":"53da590249904abef4701585f144fea80b4409b5d8bafaf36ade598da4676c91"}
Jul 28 17:10:34.489801 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.489755    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jh65p" podStartSLOduration=3.331130963 podStartE2EDuration="22.489743399s" totalImagesPullingTime="19.158612436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:34.489452388 +0000 UTC m=+22.718651504" watchObservedRunningTime="2026-07-28 17:10:34.489743399 +0000 UTC m=+22.718942494"
Jul 28 17:10:34.490174 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.490120    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-111.ec2.internal" podStartSLOduration=21.490109796 podStartE2EDuration="21.490109796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:16.462416928 +0000 UTC m=+4.691616025" watchObservedRunningTime="2026-07-28 17:10:34.490109796 +0000 UTC m=+22.719308896"
Jul 28 17:10:34.533497 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.533450    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t5xp7" podStartSLOduration=3.354452593 podStartE2EDuration="22.533435834s" totalImagesPullingTime="19.178983241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:34.533095934 +0000 UTC m=+22.762295056" watchObservedRunningTime="2026-07-28 17:10:34.533435834 +0000 UTC m=+22.762634930"
Jul 28 17:10:34.550965 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.550915    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4w8n8" podStartSLOduration=3.36864128 podStartE2EDuration="22.550893581s" totalImagesPullingTime="19.182252301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:34.549992628 +0000 UTC m=+22.779191751" watchObservedRunningTime="2026-07-28 17:10:34.550893581 +0000 UTC m=+22.780092877"
Jul 28 17:10:34.589152 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:34.589101    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f8nqb" podStartSLOduration=11.703300458 podStartE2EDuration="22.589071413s" totalImagesPullingTime="10.885770955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:34.56838538 +0000 UTC m=+22.797584479" watchObservedRunningTime="2026-07-28 17:10:34.589071413 +0000 UTC m=+22.818270510"
Jul 28 17:10:35.314404 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.314373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:35.314522 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.314374    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:35.314522 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:35.314479    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:35.314631 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.314382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:35.314631 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:35.314542    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:35.314722 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:35.314667    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:35.468775 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.468747    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:10:35.491393 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.491369    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:35.492147 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.491936    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:35.492252 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.492193    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2" event={"ID":"cdead62f-00ab-4c05-bf46-50fc2edff78d","Type":"ContainerStarted","Data":"40946ee0ae764bc9ee42607e76d13c48bb9dd8a3817b59e71a5761c2299714d6"}
Jul 28 17:10:35.493981 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.493771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k7jr9" event={"ID":"13523199-d290-434c-ba05-71c05034b7e7","Type":"ContainerStarted","Data":"7ea83a9445d133bf3d818c07bad1b7e46aaddae9bbbb7d337ba2e3ae2f15ba73"}
Jul 28 17:10:35.495792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.495773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c" event={"ID":"a6cdf093-237d-45ff-ac83-e991ebb2e51c","Type":"ContainerStarted","Data":"f5bef8a87547adba7dcb1ed8229fcf02c8b715d2819034c34c275f304a718bfc"}
Jul 28 17:10:35.496298 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.496279    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:35.496822 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.496807    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f8nqb"
Jul 28 17:10:35.508898 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.508855    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-j4jsd" podStartSLOduration=3.996464026 podStartE2EDuration="23.508842814s" totalImagesPullingTime="19.512378788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:34.589057957 +0000 UTC m=+22.818257078" watchObservedRunningTime="2026-07-28 17:10:35.508842814 +0000 UTC m=+23.738041909"
Jul 28 17:10:35.544014 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:35.543968    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k7jr9" podStartSLOduration=4.350725439 podStartE2EDuration="23.543954013s" totalImagesPullingTime="19.193228574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:35.543948348 +0000 UTC m=+23.773147456" watchObservedRunningTime="2026-07-28 17:10:35.543954013 +0000 UTC m=+23.773153109"
Jul 28 17:10:36.222051 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:36.221925    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:10:35.468766087Z","UUID":"4f42ba4d-aee4-4771-8ce0-dfc10606e287","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:10:36.223793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:36.223761    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:10:36.223936 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:36.223806    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:10:37.314111 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:37.313911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:37.314566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:37.313911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:37.314566 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:37.314195    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:37.314566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:37.313930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:37.314566 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:37.314292    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:37.314566 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:37.314394    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:37.503205 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:37.503167    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c" event={"ID":"a6cdf093-237d-45ff-ac83-e991ebb2e51c","Type":"ContainerStarted","Data":"11b228f479b44a3f2e1601f131e7cfbbfbcff2c977a2ebac8d4be3a83b0a033c"}
Jul 28 17:10:37.506686 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:37.506660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2" event={"ID":"cdead62f-00ab-4c05-bf46-50fc2edff78d","Type":"ContainerStarted","Data":"dd847fd4df6a866dfc7ce4cd98b841ec8e52fdf0c37b3fb8f19379714e6e2687"}
Jul 28 17:10:37.523754 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:37.523699    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hkk4c" podStartSLOduration=3.830537231 podStartE2EDuration="25.523679939s" totalImagesPullingTime="21.693142708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:37.522677669 +0000 UTC m=+25.751876764" watchObservedRunningTime="2026-07-28 17:10:37.523679939 +0000 UTC m=+25.752879032"
Jul 28 17:10:39.314201 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:39.314027    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:39.314825 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:39.314065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:39.314825 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:39.314280    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:39.314825 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:39.314065    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:39.314825 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:39.314335    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:39.314825 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:39.314427    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:39.512953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:39.512852    2579 generic.go:356] "Generic (PLEG): container finished" podID="76de2b66-aaa2-426f-b965-89bd9d61a207" containerID="7b7e332ec70f34b85fc02751c80db2d9bf3c0506c1c5eb82e4a0a3d9268227b7" exitCode=0
Jul 28 17:10:39.513133 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:39.512949    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57cvt" event={"ID":"76de2b66-aaa2-426f-b965-89bd9d61a207","Type":"ContainerDied","Data":"7b7e332ec70f34b85fc02751c80db2d9bf3c0506c1c5eb82e4a0a3d9268227b7"}
Jul 28 17:10:39.516537 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:39.516515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2" event={"ID":"cdead62f-00ab-4c05-bf46-50fc2edff78d","Type":"ContainerStarted","Data":"286da6880dac9ccec23472ffd012a23a91e1971d6d6ff81e9317bfeccc1acece"}
Jul 28 17:10:39.516774 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:39.516750    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:39.516834 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:39.516819    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:39.531638 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:39.531618    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:39.573198 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:39.573151    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2" podStartSLOduration=8.345496631 podStartE2EDuration="27.573136725s" totalImagesPullingTime="19.227640094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:39.572703522 +0000 UTC m=+27.801902617" watchObservedRunningTime="2026-07-28 17:10:39.573136725 +0000 UTC m=+27.802335821"
Jul 28 17:10:40.518867 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:40.518837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:40.532979 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:40.532953    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:10:40.868820 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:40.868790    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x8kpv"]
Jul 28 17:10:40.868964 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:40.868931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:40.869056 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:40.869035    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:40.871566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:40.871545    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bx6mn"]
Jul 28 17:10:40.871694 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:40.871653    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:40.871798 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:40.871756    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:40.874421 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:40.874402    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qr9ct"]
Jul 28 17:10:40.874513 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:40.874491    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:40.874594 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:40.874575    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:41.521972 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:41.521937    2579 generic.go:356] "Generic (PLEG): container finished" podID="76de2b66-aaa2-426f-b965-89bd9d61a207" containerID="713e7b2237b2004fedb3fcaf43809b1aed3105b78e1b2ff9c29b771737e865e9" exitCode=0
Jul 28 17:10:41.522371 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:41.522019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57cvt" event={"ID":"76de2b66-aaa2-426f-b965-89bd9d61a207","Type":"ContainerDied","Data":"713e7b2237b2004fedb3fcaf43809b1aed3105b78e1b2ff9c29b771737e865e9"}
Jul 28 17:10:42.315170 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:42.315143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:42.315332 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:42.315226    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:42.315332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:42.315254    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:42.315402 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:42.315349    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:42.315402 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:42.315395    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:42.315467 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:42.315440    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:43.529052 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:43.528869    2579 generic.go:356] "Generic (PLEG): container finished" podID="76de2b66-aaa2-426f-b965-89bd9d61a207" containerID="594c034654bfab5881e9d86bc0cef04a30818e1ab1713aa9f866b8bc111eabf6" exitCode=0
Jul 28 17:10:43.529052 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:43.528959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57cvt" event={"ID":"76de2b66-aaa2-426f-b965-89bd9d61a207","Type":"ContainerDied","Data":"594c034654bfab5881e9d86bc0cef04a30818e1ab1713aa9f866b8bc111eabf6"}
Jul 28 17:10:44.314148 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:44.314117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:44.314377 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:44.314117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:44.314377 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:44.314225    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:44.314377 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:44.314239    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:44.314377 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:44.314349    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:44.314588 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:44.314430    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:45.916297 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:45.916260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:45.916800 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:45.916433    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:45.916800 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:45.916505    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret podName:29dafc6b-d587-47f3-9ea9-f10cacb103fe nodeName:}" failed. No retries permitted until 2026-07-28 17:11:17.916488656 +0000 UTC m=+66.145687735 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret") pod "global-pull-secret-syncer-x8kpv" (UID: "29dafc6b-d587-47f3-9ea9-f10cacb103fe") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:10:46.016804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.016765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:46.016960 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.016927    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:46.017035 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.016989    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs podName:19846f2a-05eb-485f-8ae6-67e7090bc346 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:18.016969255 +0000 UTC m=+66.246168329 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs") pod "network-metrics-daemon-bx6mn" (UID: "19846f2a-05eb-485f-8ae6-67e7090bc346") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:10:46.117582 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.117544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8np\" (UniqueName: \"kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np\") pod \"network-check-target-qr9ct\" (UID: \"0569b22a-fdc0-4877-8d81-6b2f9debea9c\") " pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:46.117753 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.117682    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:10:46.117753 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.117696    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:10:46.117753 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.117705    2579 projected.go:196] Error preparing data for projected volume kube-api-access-lh8np for pod openshift-network-diagnostics/network-check-target-qr9ct: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:46.117916 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.117759    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np podName:0569b22a-fdc0-4877-8d81-6b2f9debea9c nodeName:}" failed. No retries permitted until 2026-07-28 17:11:18.117744148 +0000 UTC m=+66.346943222 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lh8np" (UniqueName: "kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np") pod "network-check-target-qr9ct" (UID: "0569b22a-fdc0-4877-8d81-6b2f9debea9c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:10:46.313814 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.313719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:46.313966 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.313719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:46.313966 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.313843    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bx6mn" podUID="19846f2a-05eb-485f-8ae6-67e7090bc346"
Jul 28 17:10:46.313966 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.313870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:46.314140 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.313961    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x8kpv" podUID="29dafc6b-d587-47f3-9ea9-f10cacb103fe"
Jul 28 17:10:46.314140 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.314045    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qr9ct" podUID="0569b22a-fdc0-4877-8d81-6b2f9debea9c"
Jul 28 17:10:46.615240 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.615209    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-111.ec2.internal" event="NodeReady"
Jul 28 17:10:46.615396 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.615366    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:10:46.652950 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.652917    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5ffb6d446d-nb8db"]
Jul 28 17:10:46.655665 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.655641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.659696 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.659667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:10:46.661531 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.661511    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:10:46.665809 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.665787    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:10:46.666450 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.666418    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xks97\""
Jul 28 17:10:46.671233 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.671212    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:10:46.674421 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.674403    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-clksn"]
Jul 28 17:10:46.676483 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.676465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.677461 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.677443    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-48qh4"]
Jul 28 17:10:46.679277 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.679258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-48qh4"
Jul 28 17:10:46.680575 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.680554    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:10:46.680842 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.680822    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kslsq\""
Jul 28 17:10:46.680963 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.680926    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:10:46.681738 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.681717    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ffb6d446d-nb8db"]
Jul 28 17:10:46.681884 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.681868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:10:46.681946 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.681910    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:10:46.682565 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.682362    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:10:46.682714 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.682618    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-65jvx\""
Jul 28 17:10:46.690694 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.690675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-clksn"]
Jul 28 17:10:46.694133 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.694114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-48qh4"]
Jul 28 17:10:46.823237 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-bound-sa-token\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.823435 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-image-registry-private-configuration\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.823435 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823324    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-trusted-ca\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.823435 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.823435 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7068fb4a-e8f2-4262-9873-da9abde135ac-ca-trust-extracted\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.823435 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823422    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t56kz\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-kube-api-access-t56kz\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.823652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0767256-2cea-499e-8c75-568595e72e3d-cert\") pod \"ingress-canary-48qh4\" (UID: \"a0767256-2cea-499e-8c75-568595e72e3d\") " pod="openshift-ingress-canary/ingress-canary-48qh4"
Jul 28 17:10:46.823652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823540    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-installation-pull-secrets\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.823652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fb7j\" (UniqueName: \"kubernetes.io/projected/a0767256-2cea-499e-8c75-568595e72e3d-kube-api-access-7fb7j\") pod \"ingress-canary-48qh4\" (UID: \"a0767256-2cea-499e-8c75-568595e72e3d\") " pod="openshift-ingress-canary/ingress-canary-48qh4"
Jul 28 17:10:46.823775 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823689    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-config-volume\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.823775 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-tmp-dir\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.823866 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-certificates\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.823866 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.823866 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.823816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45tb7\" (UniqueName: \"kubernetes.io/projected/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-kube-api-access-45tb7\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.924514 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.924434    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t56kz\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-kube-api-access-t56kz\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.924514 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.924476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0767256-2cea-499e-8c75-568595e72e3d-cert\") pod \"ingress-canary-48qh4\" (UID: \"a0767256-2cea-499e-8c75-568595e72e3d\") " pod="openshift-ingress-canary/ingress-canary-48qh4"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.924614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-installation-pull-secrets\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.924695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fb7j\" (UniqueName: \"kubernetes.io/projected/a0767256-2cea-499e-8c75-568595e72e3d-kube-api-access-7fb7j\") pod \"ingress-canary-48qh4\" (UID: \"a0767256-2cea-499e-8c75-568595e72e3d\") " pod="openshift-ingress-canary/ingress-canary-48qh4"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.924754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-config-volume\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.924856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-tmp-dir\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.924900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-certificates\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.924930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.924954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45tb7\" (UniqueName: \"kubernetes.io/projected/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-kube-api-access-45tb7\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.924989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-bound-sa-token\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.925018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-image-registry-private-configuration\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.925043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-trusted-ca\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.925066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.925112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.925105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7068fb4a-e8f2-4262-9873-da9abde135ac-ca-trust-extracted\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.925626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.925337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-config-volume\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.925626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.925437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7068fb4a-e8f2-4262-9873-da9abde135ac-ca-trust-extracted\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.925626 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.925538    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:46.925626 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.925552    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffb6d446d-nb8db: secret "image-registry-tls" not found
Jul 28 17:10:46.925626 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.925612    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls podName:7068fb4a-e8f2-4262-9873-da9abde135ac nodeName:}" failed. No retries permitted until 2026-07-28 17:10:47.425594618 +0000 UTC m=+35.654793707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls") pod "image-registry-5ffb6d446d-nb8db" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac") : secret "image-registry-tls" not found
Jul 28 17:10:46.925626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.925609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-certificates\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.925880 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.925848    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-tmp-dir\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.925880 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.925873    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:46.925964 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:46.925927    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls podName:f1338c56-b7f9-4bec-b8a7-8dc14677efe3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:47.425907971 +0000 UTC m=+35.655107049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls") pod "dns-default-clksn" (UID: "f1338c56-b7f9-4bec-b8a7-8dc14677efe3") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:46.926173 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.926153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-trusted-ca\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.929328 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.929304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-installation-pull-secrets\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.929423 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.929305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-image-registry-private-configuration\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.929423 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.929313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0767256-2cea-499e-8c75-568595e72e3d-cert\") pod \"ingress-canary-48qh4\" (UID: \"a0767256-2cea-499e-8c75-568595e72e3d\") " pod="openshift-ingress-canary/ingress-canary-48qh4"
Jul 28 17:10:46.937738 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.937696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-bound-sa-token\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.938023 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.937997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fb7j\" (UniqueName: \"kubernetes.io/projected/a0767256-2cea-499e-8c75-568595e72e3d-kube-api-access-7fb7j\") pod \"ingress-canary-48qh4\" (UID: \"a0767256-2cea-499e-8c75-568595e72e3d\") " pod="openshift-ingress-canary/ingress-canary-48qh4"
Jul 28 17:10:46.938341 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.938323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t56kz\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-kube-api-access-t56kz\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:46.945290 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.945249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45tb7\" (UniqueName: \"kubernetes.io/projected/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-kube-api-access-45tb7\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:46.995611 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:46.995586    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-48qh4"
Jul 28 17:10:47.429808 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:47.429773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:47.429987 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:47.429823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:47.429987 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:47.429922    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:47.429987 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:47.429932    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffb6d446d-nb8db: secret "image-registry-tls" not found
Jul 28 17:10:47.429987 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:47.429930    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:47.429987 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:47.429989    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls podName:7068fb4a-e8f2-4262-9873-da9abde135ac nodeName:}" failed. No retries permitted until 2026-07-28 17:10:48.42997643 +0000 UTC m=+36.659175503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls") pod "image-registry-5ffb6d446d-nb8db" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac") : secret "image-registry-tls" not found
Jul 28 17:10:47.430200 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:47.430002    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls podName:f1338c56-b7f9-4bec-b8a7-8dc14677efe3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:48.42999572 +0000 UTC m=+36.659194794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls") pod "dns-default-clksn" (UID: "f1338c56-b7f9-4bec-b8a7-8dc14677efe3") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:47.571900 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:47.571873    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-48qh4"]
Jul 28 17:10:47.575219 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:10:47.575190    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0767256_2cea_499e_8c75_568595e72e3d.slice/crio-d9b39106e8f623d81dedb3c45efc416e9bddf13cf161f545452275e8f13bd008 WatchSource:0}: Error finding container d9b39106e8f623d81dedb3c45efc416e9bddf13cf161f545452275e8f13bd008: Status 404 returned error can't find the container with id d9b39106e8f623d81dedb3c45efc416e9bddf13cf161f545452275e8f13bd008
Jul 28 17:10:48.313997 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.313966    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:10:48.314604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.313967    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:10:48.314604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.313966    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:10:48.316987 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.316968    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5l79z\""
Jul 28 17:10:48.318104 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.318089    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:10:48.318249 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.318235    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:10:48.320369 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.320350    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8c6kz\""
Jul 28 17:10:48.320898 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.320660    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:10:48.320898 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.320691    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:10:48.437708 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.437676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:48.437857 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.437716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:48.437857 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:48.437782    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:48.437857 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:48.437791    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:48.437857 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:48.437804    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffb6d446d-nb8db: secret "image-registry-tls" not found
Jul 28 17:10:48.437857 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:48.437846    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls podName:f1338c56-b7f9-4bec-b8a7-8dc14677efe3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:50.437828789 +0000 UTC m=+38.667027867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls") pod "dns-default-clksn" (UID: "f1338c56-b7f9-4bec-b8a7-8dc14677efe3") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:48.438028 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:48.437860    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls podName:7068fb4a-e8f2-4262-9873-da9abde135ac nodeName:}" failed. No retries permitted until 2026-07-28 17:10:50.437854777 +0000 UTC m=+38.667053851 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls") pod "image-registry-5ffb6d446d-nb8db" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac") : secret "image-registry-tls" not found
Jul 28 17:10:48.544352 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.544319    2579 generic.go:356] "Generic (PLEG): container finished" podID="76de2b66-aaa2-426f-b965-89bd9d61a207" containerID="a3574bce3dfcbdfa4035f764f95a9fbbb206132606148787b6e00342cbd2e45d" exitCode=0
Jul 28 17:10:48.544526 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.544432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57cvt" event={"ID":"76de2b66-aaa2-426f-b965-89bd9d61a207","Type":"ContainerDied","Data":"a3574bce3dfcbdfa4035f764f95a9fbbb206132606148787b6e00342cbd2e45d"}
Jul 28 17:10:48.545873 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:48.545848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-48qh4" event={"ID":"a0767256-2cea-499e-8c75-568595e72e3d","Type":"ContainerStarted","Data":"d9b39106e8f623d81dedb3c45efc416e9bddf13cf161f545452275e8f13bd008"}
Jul 28 17:10:49.552158 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:49.552113    2579 generic.go:356] "Generic (PLEG): container finished" podID="76de2b66-aaa2-426f-b965-89bd9d61a207" containerID="27a95bc0ce1cf3a8dab34c0a5fc81363aa5d2c3a6df5c380f873fb1de81eaa6e" exitCode=0
Jul 28 17:10:49.552158 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:49.552168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57cvt" event={"ID":"76de2b66-aaa2-426f-b965-89bd9d61a207","Type":"ContainerDied","Data":"27a95bc0ce1cf3a8dab34c0a5fc81363aa5d2c3a6df5c380f873fb1de81eaa6e"}
Jul 28 17:10:50.452451 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:50.452414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:50.452569 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:50.452474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:50.452649 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:50.452570    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:50.452649 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:50.452592    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffb6d446d-nb8db: secret "image-registry-tls" not found
Jul 28 17:10:50.452649 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:50.452571    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:50.452649 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:50.452641    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls podName:7068fb4a-e8f2-4262-9873-da9abde135ac nodeName:}" failed. No retries permitted until 2026-07-28 17:10:54.45262555 +0000 UTC m=+42.681824623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls") pod "image-registry-5ffb6d446d-nb8db" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac") : secret "image-registry-tls" not found
Jul 28 17:10:50.452787 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:50.452654    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls podName:f1338c56-b7f9-4bec-b8a7-8dc14677efe3 nodeName:}" failed. No retries permitted until 2026-07-28 17:10:54.452648098 +0000 UTC m=+42.681847172 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls") pod "dns-default-clksn" (UID: "f1338c56-b7f9-4bec-b8a7-8dc14677efe3") : secret "dns-default-metrics-tls" not found
Jul 28 17:10:50.557706 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:50.557548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57cvt" event={"ID":"76de2b66-aaa2-426f-b965-89bd9d61a207","Type":"ContainerStarted","Data":"d38d6333b8f2be2e7a9e63b6e801b9148976bc435f7dc337bb95a9838a3da2a5"}
Jul 28 17:10:50.591693 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:50.591646    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-57cvt" podStartSLOduration=8.772288658 podStartE2EDuration="38.591630322s" totalImagesPullingTime="29.819341664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:50.589589122 +0000 UTC m=+38.818788217" watchObservedRunningTime="2026-07-28 17:10:50.591630322 +0000 UTC m=+38.820829417"
Jul 28 17:10:51.561377 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:51.561342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-48qh4" event={"ID":"a0767256-2cea-499e-8c75-568595e72e3d","Type":"ContainerStarted","Data":"354e2fa05ecd05707923c6866272ebdbf06dcf76d6f4cd0525ebf60ad93f9782"}
Jul 28 17:10:51.582985 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:51.582940    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-48qh4" podStartSLOduration=2.686039293 podStartE2EDuration="5.582924921s" totalImagesPullingTime="2.896885628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:10:51.581927556 +0000 UTC m=+39.811126653" watchObservedRunningTime="2026-07-28 17:10:51.582924921 +0000 UTC m=+39.812124017"
Jul 28 17:10:54.479546 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:54.479506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:10:54.479951 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:10:54.479604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:10:54.479951 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:54.479681    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:10:54.479951 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:54.479698    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffb6d446d-nb8db: secret "image-registry-tls" not found
Jul 28 17:10:54.479951 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:54.479749    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls podName:7068fb4a-e8f2-4262-9873-da9abde135ac nodeName:}" failed. No retries permitted until 2026-07-28 17:11:02.479734154 +0000 UTC m=+50.708933228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls") pod "image-registry-5ffb6d446d-nb8db" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac") : secret "image-registry-tls" not found
Jul 28 17:10:54.479951 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:54.479686    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:10:54.479951 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:10:54.479799    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls podName:f1338c56-b7f9-4bec-b8a7-8dc14677efe3 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:02.479786042 +0000 UTC m=+50.708985120 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls") pod "dns-default-clksn" (UID: "f1338c56-b7f9-4bec-b8a7-8dc14677efe3") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:02.538217 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:02.538175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:11:02.538217 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:02.538203    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:02.538713 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:02.538246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:11:02.538713 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:02.538259    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls podName:f1338c56-b7f9-4bec-b8a7-8dc14677efe3 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:18.538245624 +0000 UTC m=+66.767444698 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls") pod "dns-default-clksn" (UID: "f1338c56-b7f9-4bec-b8a7-8dc14677efe3") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:02.538713 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:02.538317    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:02.538713 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:02.538327    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffb6d446d-nb8db: secret "image-registry-tls" not found
Jul 28 17:11:02.538713 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:02.538374    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls podName:7068fb4a-e8f2-4262-9873-da9abde135ac nodeName:}" failed. No retries permitted until 2026-07-28 17:11:18.53836326 +0000 UTC m=+66.767562334 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls") pod "image-registry-5ffb6d446d-nb8db" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac") : secret "image-registry-tls" not found
Jul 28 17:11:07.924316 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.924277    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24"]
Jul 28 17:11:07.926553 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.926533    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"]
Jul 28 17:11:07.926671 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.926644    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24"
Jul 28 17:11:07.929520 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.929500    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 17:11:07.929637 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.929564    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"]
Jul 28 17:11:07.929746 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.929719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:07.929903 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.929756    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:11:07.929993 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.929978    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vm7tw\""
Jul 28 17:11:07.931824 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.931804    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zxbld"]
Jul 28 17:11:07.931922 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.931907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:07.932188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.932170    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 17:11:07.932549 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.932534    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:11:07.932684 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.932671    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 17:11:07.932773 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.932759    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6js8k\""
Jul 28 17:11:07.934020 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.934003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:07.934323 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.934305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:11:07.934528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.934515    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 17:11:07.934653 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.934641    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 17:11:07.934772 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.934754    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-k5bkb\""
Jul 28 17:11:07.935056 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.935041    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 17:11:07.937006 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.936991    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6sw8p\""
Jul 28 17:11:07.937100 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.937033    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 17:11:07.937100 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.937058    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 17:11:07.937249 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.937212    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:11:07.937341 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.937325    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:11:07.942251 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.940504    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24"]
Jul 28 17:11:07.942251 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.941161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"]
Jul 28 17:11:07.942251 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.942210    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"]
Jul 28 17:11:07.943244 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.943227    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 17:11:07.952685 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.952666    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zxbld"]
Jul 28 17:11:07.976691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38193ea1-7cb0-4c1e-9664-a9b02b706559-serving-cert\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:07.976814 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfmzd\" (UniqueName: \"kubernetes.io/projected/e62fc7a6-3156-4e2d-a376-1a753ef8eb8d-kube-api-access-sfmzd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n\" (UID: \"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:07.976814 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38193ea1-7cb0-4c1e-9664-a9b02b706559-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:07.976814 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:07.976814 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czxpb\" (UniqueName: \"kubernetes.io/projected/8cb83001-d5e0-4373-90bb-15a82271db50-kube-api-access-czxpb\") pod \"volume-data-source-validator-549c6d54cf-ddr24\" (UID: \"8cb83001-d5e0-4373-90bb-15a82271db50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24"
Jul 28 17:11:07.976814 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38193ea1-7cb0-4c1e-9664-a9b02b706559-service-ca-bundle\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:07.976982 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38193ea1-7cb0-4c1e-9664-a9b02b706559-tmp\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:07.976982 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78s4d\" (UniqueName: \"kubernetes.io/projected/38193ea1-7cb0-4c1e-9664-a9b02b706559-kube-api-access-78s4d\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:07.976982 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e62fc7a6-3156-4e2d-a376-1a753ef8eb8d-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n\" (UID: \"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:07.976982 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e62fc7a6-3156-4e2d-a376-1a753ef8eb8d-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n\" (UID: \"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:07.976982 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.976962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svp6c\" (UniqueName: \"kubernetes.io/projected/7b69a45c-150e-4efc-b0c3-2f32899dd00a-kube-api-access-svp6c\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:07.977161 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:07.977046    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/38193ea1-7cb0-4c1e-9664-a9b02b706559-snapshots\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.014657 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.014634    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99"]
Jul 28 17:11:08.016864 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.016851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99"
Jul 28 17:11:08.018534 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.018492    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"]
Jul 28 17:11:08.019579 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.019562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-994fw\""
Jul 28 17:11:08.020658 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.020642    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"]
Jul 28 17:11:08.020746 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.020734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.023028 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.023010    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:08.023352 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.023331    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 17:11:08.023472 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.023458    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 17:11:08.023543 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.023529    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 17:11:08.024235 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.024220    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-c87mx\""
Jul 28 17:11:08.025486 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.025469    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 17:11:08.025867 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.025851    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:11:08.027373 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.027354    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 17:11:08.027478 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.027392    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cw6ln\""
Jul 28 17:11:08.027478 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.027392    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:11:08.027620 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.027603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:11:08.029411 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.029392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99"]
Jul 28 17:11:08.033165 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.033150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"]
Jul 28 17:11:08.033862 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.033841    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"]
Jul 28 17:11:08.038224 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.038206    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5dc8cf7cff-gn2tw"]
Jul 28 17:11:08.040303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.040288    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.042751 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.042736    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 17:11:08.042858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.042843    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 17:11:08.042914 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.042869    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 17:11:08.042963 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.042918    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rvhbw\""
Jul 28 17:11:08.042963 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.042868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 17:11:08.043194 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.043178    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 17:11:08.043267 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.043208    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 17:11:08.053795 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.053773    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5dc8cf7cff-gn2tw"]
Jul 28 17:11:08.077467 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38193ea1-7cb0-4c1e-9664-a9b02b706559-tmp\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.077557 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zlcq\" (UniqueName: \"kubernetes.io/projected/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-kube-api-access-9zlcq\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:08.077557 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077512    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78s4d\" (UniqueName: \"kubernetes.io/projected/38193ea1-7cb0-4c1e-9664-a9b02b706559-kube-api-access-78s4d\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.077557 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.077726 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e62fc7a6-3156-4e2d-a376-1a753ef8eb8d-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n\" (UID: \"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:08.077726 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-default-certificate\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.077726 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5g8c\" (UniqueName: \"kubernetes.io/projected/e77175ad-f991-48f4-bfa2-b35bdbcb138e-kube-api-access-n5g8c\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.077726 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38193ea1-7cb0-4c1e-9664-a9b02b706559-service-ca-bundle\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.077726 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svp6c\" (UniqueName: \"kubernetes.io/projected/7b69a45c-150e-4efc-b0c3-2f32899dd00a-kube-api-access-svp6c\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:08.077726 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3e259f7-b62e-4579-8db5-c423186f2510-serving-cert\") pod \"service-ca-operator-5d6c75f84f-6pcqx\" (UID: \"b3e259f7-b62e-4579-8db5-c423186f2510\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.078002 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czxpb\" (UniqueName: \"kubernetes.io/projected/8cb83001-d5e0-4373-90bb-15a82271db50-kube-api-access-czxpb\") pod \"volume-data-source-validator-549c6d54cf-ddr24\" (UID: \"8cb83001-d5e0-4373-90bb-15a82271db50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24"
Jul 28 17:11:08.078002 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3e259f7-b62e-4579-8db5-c423186f2510-config\") pod \"service-ca-operator-5d6c75f84f-6pcqx\" (UID: \"b3e259f7-b62e-4579-8db5-c423186f2510\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.078002 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/38193ea1-7cb0-4c1e-9664-a9b02b706559-tmp\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.078002 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/38193ea1-7cb0-4c1e-9664-a9b02b706559-snapshots\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.078002 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.077953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfmzd\" (UniqueName: \"kubernetes.io/projected/e62fc7a6-3156-4e2d-a376-1a753ef8eb8d-kube-api-access-sfmzd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n\" (UID: \"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:08.078249 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078009    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.078249 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7pt2\" (UniqueName: \"kubernetes.io/projected/b3e259f7-b62e-4579-8db5-c423186f2510-kube-api-access-w7pt2\") pod \"service-ca-operator-5d6c75f84f-6pcqx\" (UID: \"b3e259f7-b62e-4579-8db5-c423186f2510\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.078249 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnxk\" (UniqueName: \"kubernetes.io/projected/6f2443b8-9d7d-4c24-93f1-495210dff36d-kube-api-access-nbnxk\") pod \"network-check-source-7d88cb55d8-dbf99\" (UID: \"6f2443b8-9d7d-4c24-93f1-495210dff36d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99"
Jul 28 17:11:08.078249 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e62fc7a6-3156-4e2d-a376-1a753ef8eb8d-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n\" (UID: \"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:08.078249 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38193ea1-7cb0-4c1e-9664-a9b02b706559-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.078249 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078235    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:08.078506 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078275    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38193ea1-7cb0-4c1e-9664-a9b02b706559-serving-cert\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.078506 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38193ea1-7cb0-4c1e-9664-a9b02b706559-service-ca-bundle\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.078506 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:08.078506 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-stats-auth\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.078506 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078363    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:08.078506 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.078367    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:11:08.078506 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.078436    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls podName:7b69a45c-150e-4efc-b0c3-2f32899dd00a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:08.5784182 +0000 UTC m=+56.807617286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls") pod "cluster-samples-operator-6748489656-ssvr9" (UID: "7b69a45c-150e-4efc-b0c3-2f32899dd00a") : secret "samples-operator-tls" not found
Jul 28 17:11:08.078787 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/38193ea1-7cb0-4c1e-9664-a9b02b706559-snapshots\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.078787 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078669    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e62fc7a6-3156-4e2d-a376-1a753ef8eb8d-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n\" (UID: \"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:08.078987 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.078971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/38193ea1-7cb0-4c1e-9664-a9b02b706559-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.081593 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.081577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e62fc7a6-3156-4e2d-a376-1a753ef8eb8d-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n\" (UID: \"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:08.081638 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.081622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38193ea1-7cb0-4c1e-9664-a9b02b706559-serving-cert\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.084802 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.084784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78s4d\" (UniqueName: \"kubernetes.io/projected/38193ea1-7cb0-4c1e-9664-a9b02b706559-kube-api-access-78s4d\") pod \"insights-operator-5885fdb759-zxbld\" (UID: \"38193ea1-7cb0-4c1e-9664-a9b02b706559\") " pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.087204 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.087184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svp6c\" (UniqueName: \"kubernetes.io/projected/7b69a45c-150e-4efc-b0c3-2f32899dd00a-kube-api-access-svp6c\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:08.087464 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.087445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfmzd\" (UniqueName: \"kubernetes.io/projected/e62fc7a6-3156-4e2d-a376-1a753ef8eb8d-kube-api-access-sfmzd\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n\" (UID: \"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:08.087694 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.087676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czxpb\" (UniqueName: \"kubernetes.io/projected/8cb83001-d5e0-4373-90bb-15a82271db50-kube-api-access-czxpb\") pod \"volume-data-source-validator-549c6d54cf-ddr24\" (UID: \"8cb83001-d5e0-4373-90bb-15a82271db50\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24"
Jul 28 17:11:08.116596 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.116577    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-2gspt"]
Jul 28 17:11:08.119856 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.119842    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.122636 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.122618    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 17:11:08.122725 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.122665    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:11:08.122769 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.122744    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 17:11:08.122914 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.122896    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 17:11:08.122965 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.122931    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-wl4q6\""
Jul 28 17:11:08.128511 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.128494    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 17:11:08.134279 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.134253    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-2gspt"]
Jul 28 17:11:08.179613 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.179536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3e259f7-b62e-4579-8db5-c423186f2510-serving-cert\") pod \"service-ca-operator-5d6c75f84f-6pcqx\" (UID: \"b3e259f7-b62e-4579-8db5-c423186f2510\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.179613 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.179572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72263172-706a-41e4-89ab-b2c5a52d86ff-serving-cert\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.179613 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.179605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3e259f7-b62e-4579-8db5-c423186f2510-config\") pod \"service-ca-operator-5d6c75f84f-6pcqx\" (UID: \"b3e259f7-b62e-4579-8db5-c423186f2510\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.179858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.179743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.179858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.179778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7pt2\" (UniqueName: \"kubernetes.io/projected/b3e259f7-b62e-4579-8db5-c423186f2510-kube-api-access-w7pt2\") pod \"service-ca-operator-5d6c75f84f-6pcqx\" (UID: \"b3e259f7-b62e-4579-8db5-c423186f2510\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.179858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.179804    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnxk\" (UniqueName: \"kubernetes.io/projected/6f2443b8-9d7d-4c24-93f1-495210dff36d-kube-api-access-nbnxk\") pod \"network-check-source-7d88cb55d8-dbf99\" (UID: \"6f2443b8-9d7d-4c24-93f1-495210dff36d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99"
Jul 28 17:11:08.179858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.179838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72263172-706a-41e4-89ab-b2c5a52d86ff-trusted-ca\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.180043 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.179880    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:08.180043 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.179901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72263172-706a-41e4-89ab-b2c5a52d86ff-config\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.180043 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.179947    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:08.679927553 +0000 UTC m=+56.909126642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : secret "router-metrics-certs-default" not found
Jul 28 17:11:08.180043 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.180040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:08.180277 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.180096    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-stats-auth\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.180277 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.180153    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:08.180277 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.180221    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls podName:1087a8f2-182c-40ca-95f2-9f81ce0e4bec nodeName:}" failed. No retries permitted until 2026-07-28 17:11:08.680204064 +0000 UTC m=+56.909403140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xrstz" (UID: "1087a8f2-182c-40ca-95f2-9f81ce0e4bec") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:08.180277 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.180226    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b3e259f7-b62e-4579-8db5-c423186f2510-config\") pod \"service-ca-operator-5d6c75f84f-6pcqx\" (UID: \"b3e259f7-b62e-4579-8db5-c423186f2510\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.180277 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.180154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:08.180470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.180288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zlcq\" (UniqueName: \"kubernetes.io/projected/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-kube-api-access-9zlcq\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:08.180470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.180320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.180470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.180351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-default-certificate\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.180470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.180377    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5g8c\" (UniqueName: \"kubernetes.io/projected/e77175ad-f991-48f4-bfa2-b35bdbcb138e-kube-api-access-n5g8c\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.180470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.180405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxts8\" (UniqueName: \"kubernetes.io/projected/72263172-706a-41e4-89ab-b2c5a52d86ff-kube-api-access-vxts8\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.180699 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.180470    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:08.680452997 +0000 UTC m=+56.909652081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:08.180888 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.180869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:08.182054 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.182031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b3e259f7-b62e-4579-8db5-c423186f2510-serving-cert\") pod \"service-ca-operator-5d6c75f84f-6pcqx\" (UID: \"b3e259f7-b62e-4579-8db5-c423186f2510\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.182340 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.182317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-stats-auth\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.182552 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.182534    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-default-certificate\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.189798 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.189776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7pt2\" (UniqueName: \"kubernetes.io/projected/b3e259f7-b62e-4579-8db5-c423186f2510-kube-api-access-w7pt2\") pod \"service-ca-operator-5d6c75f84f-6pcqx\" (UID: \"b3e259f7-b62e-4579-8db5-c423186f2510\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.189976 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.189959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnxk\" (UniqueName: \"kubernetes.io/projected/6f2443b8-9d7d-4c24-93f1-495210dff36d-kube-api-access-nbnxk\") pod \"network-check-source-7d88cb55d8-dbf99\" (UID: \"6f2443b8-9d7d-4c24-93f1-495210dff36d\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99"
Jul 28 17:11:08.190491 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.190471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zlcq\" (UniqueName: \"kubernetes.io/projected/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-kube-api-access-9zlcq\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:08.190535 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.190518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5g8c\" (UniqueName: \"kubernetes.io/projected/e77175ad-f991-48f4-bfa2-b35bdbcb138e-kube-api-access-n5g8c\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.237289 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.237270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24"
Jul 28 17:11:08.250589 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.250568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"
Jul 28 17:11:08.255791 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.255762    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-zxbld"
Jul 28 17:11:08.281837 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.281800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72263172-706a-41e4-89ab-b2c5a52d86ff-trusted-ca\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.281996 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.281860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72263172-706a-41e4-89ab-b2c5a52d86ff-config\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.281996 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.281933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxts8\" (UniqueName: \"kubernetes.io/projected/72263172-706a-41e4-89ab-b2c5a52d86ff-kube-api-access-vxts8\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.281996 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.281973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72263172-706a-41e4-89ab-b2c5a52d86ff-serving-cert\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.282552 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.282528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/72263172-706a-41e4-89ab-b2c5a52d86ff-config\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.282821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.282804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/72263172-706a-41e4-89ab-b2c5a52d86ff-trusted-ca\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.284187 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.284168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/72263172-706a-41e4-89ab-b2c5a52d86ff-serving-cert\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.294533 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.294505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxts8\" (UniqueName: \"kubernetes.io/projected/72263172-706a-41e4-89ab-b2c5a52d86ff-kube-api-access-vxts8\") pod \"console-operator-584dcb6b54-2gspt\" (UID: \"72263172-706a-41e4-89ab-b2c5a52d86ff\") " pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.327028 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.326846    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99"
Jul 28 17:11:08.334718 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.334695    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"
Jul 28 17:11:08.383114 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.383014    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24"]
Jul 28 17:11:08.388935 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:08.388889    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb83001_d5e0_4373_90bb_15a82271db50.slice/crio-02a5e78836c4af053a35055d401dcabf3e51d51a0f631578927783bb83a756a2 WatchSource:0}: Error finding container 02a5e78836c4af053a35055d401dcabf3e51d51a0f631578927783bb83a756a2: Status 404 returned error can't find the container with id 02a5e78836c4af053a35055d401dcabf3e51d51a0f631578927783bb83a756a2
Jul 28 17:11:08.429547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.429518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:08.469637 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.469601    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99"]
Jul 28 17:11:08.473229 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:08.473201    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2443b8_9d7d_4c24_93f1_495210dff36d.slice/crio-91b1ca57959aa1fb0f276cf85ac343574433b5e62a8f55a1ef427154b4f69459 WatchSource:0}: Error finding container 91b1ca57959aa1fb0f276cf85ac343574433b5e62a8f55a1ef427154b4f69459: Status 404 returned error can't find the container with id 91b1ca57959aa1fb0f276cf85ac343574433b5e62a8f55a1ef427154b4f69459
Jul 28 17:11:08.485566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.485539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx"]
Jul 28 17:11:08.490106 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:08.490068    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e259f7_b62e_4579_8db5_c423186f2510.slice/crio-2f1650530b2b10417dbbd66c07c409d988c36f49614fd56f9c26c67b65e8b211 WatchSource:0}: Error finding container 2f1650530b2b10417dbbd66c07c409d988c36f49614fd56f9c26c67b65e8b211: Status 404 returned error can't find the container with id 2f1650530b2b10417dbbd66c07c409d988c36f49614fd56f9c26c67b65e8b211
Jul 28 17:11:08.561429 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.561399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-2gspt"]
Jul 28 17:11:08.564295 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:08.564271    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72263172_706a_41e4_89ab_b2c5a52d86ff.slice/crio-2bdb6d146aa9fdea039268866d42df15a267425cfdc7206d9fd377c695ff858e WatchSource:0}: Error finding container 2bdb6d146aa9fdea039268866d42df15a267425cfdc7206d9fd377c695ff858e: Status 404 returned error can't find the container with id 2bdb6d146aa9fdea039268866d42df15a267425cfdc7206d9fd377c695ff858e
Jul 28 17:11:08.586477 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.586453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:08.586637 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.586616    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:11:08.586711 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.586698    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls podName:7b69a45c-150e-4efc-b0c3-2f32899dd00a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:09.586676172 +0000 UTC m=+57.815875252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls") pod "cluster-samples-operator-6748489656-ssvr9" (UID: "7b69a45c-150e-4efc-b0c3-2f32899dd00a") : secret "samples-operator-tls" not found
Jul 28 17:11:08.599786 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.599763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24" event={"ID":"8cb83001-d5e0-4373-90bb-15a82271db50","Type":"ContainerStarted","Data":"02a5e78836c4af053a35055d401dcabf3e51d51a0f631578927783bb83a756a2"}
Jul 28 17:11:08.600800 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.600773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99" event={"ID":"6f2443b8-9d7d-4c24-93f1-495210dff36d","Type":"ContainerStarted","Data":"91b1ca57959aa1fb0f276cf85ac343574433b5e62a8f55a1ef427154b4f69459"}
Jul 28 17:11:08.601827 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.601802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt" event={"ID":"72263172-706a-41e4-89ab-b2c5a52d86ff","Type":"ContainerStarted","Data":"2bdb6d146aa9fdea039268866d42df15a267425cfdc7206d9fd377c695ff858e"}
Jul 28 17:11:08.602802 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.602785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx" event={"ID":"b3e259f7-b62e-4579-8db5-c423186f2510","Type":"ContainerStarted","Data":"2f1650530b2b10417dbbd66c07c409d988c36f49614fd56f9c26c67b65e8b211"}
Jul 28 17:11:08.611309 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.611280    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n"]
Jul 28 17:11:08.612239 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.612222    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-zxbld"]
Jul 28 17:11:08.615589 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:08.615569    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode62fc7a6_3156_4e2d_a376_1a753ef8eb8d.slice/crio-6caf348e6ea073b5fd3e97814685d0c6190c902d2e724347bac8e1264160a521 WatchSource:0}: Error finding container 6caf348e6ea073b5fd3e97814685d0c6190c902d2e724347bac8e1264160a521: Status 404 returned error can't find the container with id 6caf348e6ea073b5fd3e97814685d0c6190c902d2e724347bac8e1264160a521
Jul 28 17:11:08.615991 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:08.615975    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38193ea1_7cb0_4c1e_9664_a9b02b706559.slice/crio-01845682850d5495cd3dcf600cd053802aba5547a432ec238c1111b50812a951 WatchSource:0}: Error finding container 01845682850d5495cd3dcf600cd053802aba5547a432ec238c1111b50812a951: Status 404 returned error can't find the container with id 01845682850d5495cd3dcf600cd053802aba5547a432ec238c1111b50812a951
Jul 28 17:11:08.687748 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.687687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.687852 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.687799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:08.687852 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.687839    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:09.687823405 +0000 UTC m=+57.917022480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:08.687952 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.687895    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:08.687952 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:08.687935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:08.687952 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.687943    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:09.687926573 +0000 UTC m=+57.917125665 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : secret "router-metrics-certs-default" not found
Jul 28 17:11:08.688096 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.688026    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:08.688096 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:08.688060    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls podName:1087a8f2-182c-40ca-95f2-9f81ce0e4bec nodeName:}" failed. No retries permitted until 2026-07-28 17:11:09.688052278 +0000 UTC m=+57.917251351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xrstz" (UID: "1087a8f2-182c-40ca-95f2-9f81ce0e4bec") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:09.596213 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:09.595545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:09.596213 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:09.595779    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:11:09.596213 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:09.595843    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls podName:7b69a45c-150e-4efc-b0c3-2f32899dd00a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:11.595824931 +0000 UTC m=+59.825024008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls") pod "cluster-samples-operator-6748489656-ssvr9" (UID: "7b69a45c-150e-4efc-b0c3-2f32899dd00a") : secret "samples-operator-tls" not found
Jul 28 17:11:09.609811 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:09.609777    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zxbld" event={"ID":"38193ea1-7cb0-4c1e-9664-a9b02b706559","Type":"ContainerStarted","Data":"01845682850d5495cd3dcf600cd053802aba5547a432ec238c1111b50812a951"}
Jul 28 17:11:09.613257 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:09.613233    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n" event={"ID":"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d","Type":"ContainerStarted","Data":"6caf348e6ea073b5fd3e97814685d0c6190c902d2e724347bac8e1264160a521"}
Jul 28 17:11:09.696766 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:09.696730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:09.696936 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:09.696832    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:09.696936 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:09.696872    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:09.697072 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:09.697030    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:11.697011233 +0000 UTC m=+59.926210310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:09.697480 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:09.697459    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:09.697578 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:09.697515    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:11.697500147 +0000 UTC m=+59.926699238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : secret "router-metrics-certs-default" not found
Jul 28 17:11:09.697649 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:09.697573    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:09.697706 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:09.697674    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls podName:1087a8f2-182c-40ca-95f2-9f81ce0e4bec nodeName:}" failed. No retries permitted until 2026-07-28 17:11:11.697661686 +0000 UTC m=+59.926860763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xrstz" (UID: "1087a8f2-182c-40ca-95f2-9f81ce0e4bec") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:10.618350 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:10.618308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24" event={"ID":"8cb83001-d5e0-4373-90bb-15a82271db50","Type":"ContainerStarted","Data":"565f19b861cbb825f6dad8acc9aace7e200f3c2cb76d680a1ce7286cd6be96e5"}
Jul 28 17:11:10.635014 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:10.634846    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-ddr24" podStartSLOduration=2.095602083 podStartE2EDuration="3.634825648s" totalImagesPullingTime="1.539223565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:10.634445456 +0000 UTC m=+58.863644574" watchObservedRunningTime="2026-07-28 17:11:10.634825648 +0000 UTC m=+58.864024745"
Jul 28 17:11:11.613344 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:11.613311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:11.613509 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:11.613476    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:11:11.613570 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:11.613542    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls podName:7b69a45c-150e-4efc-b0c3-2f32899dd00a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:15.613526005 +0000 UTC m=+63.842725079 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls") pod "cluster-samples-operator-6748489656-ssvr9" (UID: "7b69a45c-150e-4efc-b0c3-2f32899dd00a") : secret "samples-operator-tls" not found
Jul 28 17:11:11.714751 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:11.714711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:11.715246 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:11.714911    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:15.714884436 +0000 UTC m=+63.944083511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:11.715246 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:11.714995    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:11.715246 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:11.715105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:11.715246 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:11.715133    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:11.715246 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:11.715176    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:15.715164648 +0000 UTC m=+63.944363722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : secret "router-metrics-certs-default" not found
Jul 28 17:11:11.715246 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:11.715199    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:11.715246 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:11.715239    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls podName:1087a8f2-182c-40ca-95f2-9f81ce0e4bec nodeName:}" failed. No retries permitted until 2026-07-28 17:11:15.715226269 +0000 UTC m=+63.944425343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xrstz" (UID: "1087a8f2-182c-40ca-95f2-9f81ce0e4bec") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:12.538321 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:12.538289    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-b92c2"
Jul 28 17:11:14.632863 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.632831    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/0.log"
Jul 28 17:11:14.633382 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.632876    2579 generic.go:356] "Generic (PLEG): container finished" podID="72263172-706a-41e4-89ab-b2c5a52d86ff" containerID="a00852dcd9ffb79f676ccf84e996a86c7f41309482a9c9b556e93321b2255985" exitCode=255
Jul 28 17:11:14.633382 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.632959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt" event={"ID":"72263172-706a-41e4-89ab-b2c5a52d86ff","Type":"ContainerDied","Data":"a00852dcd9ffb79f676ccf84e996a86c7f41309482a9c9b556e93321b2255985"}
Jul 28 17:11:14.633382 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.633377    2579 scope.go:117] "RemoveContainer" containerID="a00852dcd9ffb79f676ccf84e996a86c7f41309482a9c9b556e93321b2255985"
Jul 28 17:11:14.634904 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.634869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zxbld" event={"ID":"38193ea1-7cb0-4c1e-9664-a9b02b706559","Type":"ContainerStarted","Data":"00e66e035d1cb751c8b0b181db76239d094c28250a42803ac8763f5d0fd7701e"}
Jul 28 17:11:14.636576 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.636548    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n" event={"ID":"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d","Type":"ContainerStarted","Data":"882b7aa797caf81eddc90ce76ca2271c43011a996882db9756cc45a56665325b"}
Jul 28 17:11:14.638398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.638380    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx" event={"ID":"b3e259f7-b62e-4579-8db5-c423186f2510","Type":"ContainerStarted","Data":"42ab1bc32113851955f6677f31397e06129407206be48365337edcc39a47c18d"}
Jul 28 17:11:14.640165 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.640112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99" event={"ID":"6f2443b8-9d7d-4c24-93f1-495210dff36d","Type":"ContainerStarted","Data":"8f73255c140048fcf75b910fcbf0b7bd429acfdc10cacb98f8bc7ce900ca6498"}
Jul 28 17:11:14.665179 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.665132    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n" podStartSLOduration=2.410685811 podStartE2EDuration="7.665114967s" totalImagesPullingTime="5.254429156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:14.664395108 +0000 UTC m=+62.893594206" watchObservedRunningTime="2026-07-28 17:11:14.665114967 +0000 UTC m=+62.894314063"
Jul 28 17:11:14.700621 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.700577    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-zxbld" podStartSLOduration=2.449750537 podStartE2EDuration="7.70055751s" totalImagesPullingTime="5.250806973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:14.682457383 +0000 UTC m=+62.911656514" watchObservedRunningTime="2026-07-28 17:11:14.70055751 +0000 UTC m=+62.929756608"
Jul 28 17:11:14.701293 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.701252    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-dbf99" podStartSLOduration=2.289685925 podStartE2EDuration="7.701236975s" totalImagesPullingTime="5.41155105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:14.700869293 +0000 UTC m=+62.930068390" watchObservedRunningTime="2026-07-28 17:11:14.701236975 +0000 UTC m=+62.930436071"
Jul 28 17:11:14.724599 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:14.724307    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx" podStartSLOduration=1.34745176 podStartE2EDuration="6.724293268s" totalImagesPullingTime="5.376841508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:14.72362376 +0000 UTC m=+62.952822858" watchObservedRunningTime="2026-07-28 17:11:14.724293268 +0000 UTC m=+62.953492368"
Jul 28 17:11:15.520171 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.519429    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fg6ls"]
Jul 28 17:11:15.522155 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.522128    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.525609 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.525585    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:11:15.526025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.526006    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rd4np\""
Jul 28 17:11:15.526667 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.526655    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:11:15.549895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.549874    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fg6ls"]
Jul 28 17:11:15.644289 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.644261    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:11:15.644706 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.644691    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/0.log"
Jul 28 17:11:15.644754 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.644725    2579 generic.go:356] "Generic (PLEG): container finished" podID="72263172-706a-41e4-89ab-b2c5a52d86ff" containerID="d2327ce288e67f41c4e91c54a1a5eb5ec650688de0b3489515dd89e68d1f31fb" exitCode=255
Jul 28 17:11:15.644791 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.644754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt" event={"ID":"72263172-706a-41e4-89ab-b2c5a52d86ff","Type":"ContainerDied","Data":"d2327ce288e67f41c4e91c54a1a5eb5ec650688de0b3489515dd89e68d1f31fb"}
Jul 28 17:11:15.644850 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.644836    2579 scope.go:117] "RemoveContainer" containerID="a00852dcd9ffb79f676ccf84e996a86c7f41309482a9c9b556e93321b2255985"
Jul 28 17:11:15.645258 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.645194    2579 scope.go:117] "RemoveContainer" containerID="d2327ce288e67f41c4e91c54a1a5eb5ec650688de0b3489515dd89e68d1f31fb"
Jul 28 17:11:15.645413 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.645391    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-2gspt_openshift-console-operator(72263172-706a-41e4-89ab-b2c5a52d86ff)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt" podUID="72263172-706a-41e4-89ab-b2c5a52d86ff"
Jul 28 17:11:15.654502 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.654481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:15.654596 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.654531    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7nwn\" (UniqueName: \"kubernetes.io/projected/b88bc623-3928-4437-9acc-a10ecec23114-kube-api-access-z7nwn\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.654596 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.654570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.654679 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.654611    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 17:11:15.654679 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.654621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b88bc623-3928-4437-9acc-a10ecec23114-crio-socket\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.654679 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.654663    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls podName:7b69a45c-150e-4efc-b0c3-2f32899dd00a nodeName:}" failed. No retries permitted until 2026-07-28 17:11:23.654649553 +0000 UTC m=+71.883848627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls") pod "cluster-samples-operator-6748489656-ssvr9" (UID: "7b69a45c-150e-4efc-b0c3-2f32899dd00a") : secret "samples-operator-tls" not found
Jul 28 17:11:15.654791 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.654725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b88bc623-3928-4437-9acc-a10ecec23114-data-volume\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.654791 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.654767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.755698 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.755664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b88bc623-3928-4437-9acc-a10ecec23114-crio-socket\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.755822 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.755751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b88bc623-3928-4437-9acc-a10ecec23114-crio-socket\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.755876 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.755823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b88bc623-3928-4437-9acc-a10ecec23114-data-volume\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.755876 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.755855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:15.755955 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.755939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.756114 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.756072    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:15.756197 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.756111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:15.756197 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.756154    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7nwn\" (UniqueName: \"kubernetes.io/projected/b88bc623-3928-4437-9acc-a10ecec23114-kube-api-access-z7nwn\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.756197 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.756170    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:23.756150983 +0000 UTC m=+71.985350079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : secret "router-metrics-certs-default" not found
Jul 28 17:11:15.756347 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.756215    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b88bc623-3928-4437-9acc-a10ecec23114-data-volume\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.756347 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.756299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:15.756458 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.756384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:15.756517 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.756470    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:23.756454308 +0000 UTC m=+71.985653383 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:15.756517 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.756504    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:15.756517 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.756509    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:15.756662 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.756542    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:15.756662 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.756606    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:16.256588018 +0000 UTC m=+64.485787096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:15.756662 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.756631    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls podName:1087a8f2-182c-40ca-95f2-9f81ce0e4bec nodeName:}" failed. No retries permitted until 2026-07-28 17:11:23.756615666 +0000 UTC m=+71.985814761 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xrstz" (UID: "1087a8f2-182c-40ca-95f2-9f81ce0e4bec") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:15.756662 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:15.756654    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:16.256642222 +0000 UTC m=+64.485841299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:15.765260 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:15.765237    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7nwn\" (UniqueName: \"kubernetes.io/projected/b88bc623-3928-4437-9acc-a10ecec23114-kube-api-access-z7nwn\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:16.261150 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:16.261119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:16.261322 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:16.261188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:16.261322 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:16.261241    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:16.261322 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:16.261321    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:17.261300685 +0000 UTC m=+65.490499763 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:16.261432 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:16.261338    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:16.261432 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:16.261397    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:17.261385592 +0000 UTC m=+65.490584678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:16.649442 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:16.649419    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:11:16.649787 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:16.649730    2579 scope.go:117] "RemoveContainer" containerID="d2327ce288e67f41c4e91c54a1a5eb5ec650688de0b3489515dd89e68d1f31fb"
Jul 28 17:11:16.649891 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:16.649872    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-2gspt_openshift-console-operator(72263172-706a-41e4-89ab-b2c5a52d86ff)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt" podUID="72263172-706a-41e4-89ab-b2c5a52d86ff"
Jul 28 17:11:17.270346 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:17.270308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:17.270540 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:17.270439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:17.270540 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:17.270456    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:17.270540 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:17.270510    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:19.270495275 +0000 UTC m=+67.499694349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:17.270710 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:17.270548    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:17.270710 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:17.270603    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:19.270588601 +0000 UTC m=+67.499787674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:17.321209 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:17.321187    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jh65p_34d3b868-e31e-4fd6-9b00-f117a39ebbc3/dns-node-resolver/0.log"
Jul 28 17:11:17.929966 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:17.929940    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4w8n8_e925228f-d166-4239-8f86-dad787c3e6ee/node-ca/0.log"
Jul 28 17:11:17.975600 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:17.975571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:11:17.978211 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:17.978196    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:11:17.988488 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:17.988466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29dafc6b-d587-47f3-9ea9-f10cacb103fe-original-pull-secret\") pod \"global-pull-secret-syncer-x8kpv\" (UID: \"29dafc6b-d587-47f3-9ea9-f10cacb103fe\") " pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:11:18.029604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.029579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x8kpv"
Jul 28 17:11:18.076407 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.076376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:11:18.079054 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.079031    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:11:18.087159 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:18.087135    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:11:18.087292 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:18.087214    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs podName:19846f2a-05eb-485f-8ae6-67e7090bc346 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:22.087192997 +0000 UTC m=+130.316392087 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs") pod "network-metrics-daemon-bx6mn" (UID: "19846f2a-05eb-485f-8ae6-67e7090bc346") : secret "metrics-daemon-secret" not found
Jul 28 17:11:18.150394 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.150364    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x8kpv"]
Jul 28 17:11:18.153561 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:18.153451    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dafc6b_d587_47f3_9ea9_f10cacb103fe.slice/crio-e4fdb094af809720183cfb027a4ed9dc89813fc7377dd37c0a8ab05fb4c10fcc WatchSource:0}: Error finding container e4fdb094af809720183cfb027a4ed9dc89813fc7377dd37c0a8ab05fb4c10fcc: Status 404 returned error can't find the container with id e4fdb094af809720183cfb027a4ed9dc89813fc7377dd37c0a8ab05fb4c10fcc
Jul 28 17:11:18.177746 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.177724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh8np\" (UniqueName: \"kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np\") pod \"network-check-target-qr9ct\" (UID: \"0569b22a-fdc0-4877-8d81-6b2f9debea9c\") " pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:11:18.180020 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.179978    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh8np\" (UniqueName: \"kubernetes.io/projected/0569b22a-fdc0-4877-8d81-6b2f9debea9c-kube-api-access-lh8np\") pod \"network-check-target-qr9ct\" (UID: \"0569b22a-fdc0-4877-8d81-6b2f9debea9c\") " pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:11:18.337662 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.337629    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5l79z\""
Jul 28 17:11:18.346074 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.346052    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:11:18.430150 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.430073    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:18.430150 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.430149    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:18.430543 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.430526    2579 scope.go:117] "RemoveContainer" containerID="d2327ce288e67f41c4e91c54a1a5eb5ec650688de0b3489515dd89e68d1f31fb"
Jul 28 17:11:18.430725 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:18.430703    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-2gspt_openshift-console-operator(72263172-706a-41e4-89ab-b2c5a52d86ff)\"" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt" podUID="72263172-706a-41e4-89ab-b2c5a52d86ff"
Jul 28 17:11:18.515650 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.515619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qr9ct"]
Jul 28 17:11:18.518658 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:18.518633    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0569b22a_fdc0_4877_8d81_6b2f9debea9c.slice/crio-93818d7c26a6130d27f2161d3e9ee70750c5facf3c57f323a5f370637e1350a4 WatchSource:0}: Error finding container 93818d7c26a6130d27f2161d3e9ee70750c5facf3c57f323a5f370637e1350a4: Status 404 returned error can't find the container with id 93818d7c26a6130d27f2161d3e9ee70750c5facf3c57f323a5f370637e1350a4
Jul 28 17:11:18.581838 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.581810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:11:18.581935 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.581915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:11:18.582003 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:18.581960    2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:11:18.582003 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:18.581977    2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5ffb6d446d-nb8db: secret "image-registry-tls" not found
Jul 28 17:11:18.582071 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:18.582041    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls podName:7068fb4a-e8f2-4262-9873-da9abde135ac nodeName:}" failed. No retries permitted until 2026-07-28 17:11:50.582020377 +0000 UTC m=+98.811219464 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls") pod "image-registry-5ffb6d446d-nb8db" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac") : secret "image-registry-tls" not found
Jul 28 17:11:18.582071 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:18.582044    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:11:18.582192 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:18.582121    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls podName:f1338c56-b7f9-4bec-b8a7-8dc14677efe3 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:50.582104734 +0000 UTC m=+98.811303815 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls") pod "dns-default-clksn" (UID: "f1338c56-b7f9-4bec-b8a7-8dc14677efe3") : secret "dns-default-metrics-tls" not found
Jul 28 17:11:18.656486 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.656455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qr9ct" event={"ID":"0569b22a-fdc0-4877-8d81-6b2f9debea9c","Type":"ContainerStarted","Data":"9c3c9b7faf861dd734010df308e7beb39022a334c8a128be2cc67533f7fe3fbe"}
Jul 28 17:11:18.656586 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.656507    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qr9ct" event={"ID":"0569b22a-fdc0-4877-8d81-6b2f9debea9c","Type":"ContainerStarted","Data":"93818d7c26a6130d27f2161d3e9ee70750c5facf3c57f323a5f370637e1350a4"}
Jul 28 17:11:18.656586 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.656543    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:11:18.657594 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.657576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x8kpv" event={"ID":"29dafc6b-d587-47f3-9ea9-f10cacb103fe","Type":"ContainerStarted","Data":"e4fdb094af809720183cfb027a4ed9dc89813fc7377dd37c0a8ab05fb4c10fcc"}
Jul 28 17:11:18.676730 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.676677    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qr9ct" podStartSLOduration=66.676659294 podStartE2EDuration="1m6.676659294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:18.675049842 +0000 UTC m=+66.904248939" watchObservedRunningTime="2026-07-28 17:11:18.676659294 +0000 UTC m=+66.905858391"
Jul 28 17:11:18.721950 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:18.721886    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-48qh4_a0767256-2cea-499e-8c75-568595e72e3d/serve-healthcheck-canary/0.log"
Jul 28 17:11:19.288924 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:19.288888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:19.289382 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:19.289000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:19.289382 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:19.289024    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:19.289382 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:19.289119    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:23.289098773 +0000 UTC m=+71.518297862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:19.289382 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:19.289150    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:19.289382 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:19.289203    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:23.289188238 +0000 UTC m=+71.518387312 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:22.674169 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:22.674124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x8kpv" event={"ID":"29dafc6b-d587-47f3-9ea9-f10cacb103fe","Type":"ContainerStarted","Data":"1743dbc4b1c0a96529d6df035db7b91ed2b16f66b5b185e61b8d17a4afcac2c0"}
Jul 28 17:11:22.701090 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:22.701035    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x8kpv" podStartSLOduration=67.025480237 podStartE2EDuration="1m10.701021016s" totalImagesPullingTime="3.675540779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:22.700179379 +0000 UTC m=+70.929378495" watchObservedRunningTime="2026-07-28 17:11:22.701021016 +0000 UTC m=+70.930220147"
Jul 28 17:11:23.326072 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:23.326032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:23.326268 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:23.326141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:23.326268 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:23.326158    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:11:23.326268 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:23.326237    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:31.326220607 +0000 UTC m=+79.555419681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : configmap "kube-rbac-proxy" not found
Jul 28 17:11:23.326268 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:23.326258    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:23.326418 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:23.326311    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:31.326298347 +0000 UTC m=+79.555497421 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:23.730620 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:23.730535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:23.733133 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:23.733100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7b69a45c-150e-4efc-b0c3-2f32899dd00a-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-ssvr9\" (UID: \"7b69a45c-150e-4efc-b0c3-2f32899dd00a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:23.831026 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:23.830988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:23.831236 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:23.831098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:23.831236 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:23.831138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:23.831236 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:23.831165    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:11:23.831236 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:23.831217    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:23.831236 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:23.831240    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:39.8312196 +0000 UTC m=+88.060418688 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : secret "router-metrics-certs-default" not found
Jul 28 17:11:23.831491 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:23.831259    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls podName:1087a8f2-182c-40ca-95f2-9f81ce0e4bec nodeName:}" failed. No retries permitted until 2026-07-28 17:11:39.831247959 +0000 UTC m=+88.060447033 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-xrstz" (UID: "1087a8f2-182c-40ca-95f2-9f81ce0e4bec") : secret "cluster-monitoring-operator-tls" not found
Jul 28 17:11:23.831491 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:23.831272    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle podName:e77175ad-f991-48f4-bfa2-b35bdbcb138e nodeName:}" failed. No retries permitted until 2026-07-28 17:11:39.831266361 +0000 UTC m=+88.060465435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle") pod "router-default-5dc8cf7cff-gn2tw" (UID: "e77175ad-f991-48f4-bfa2-b35bdbcb138e") : configmap references non-existent config key: service-ca.crt
Jul 28 17:11:23.847790 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:23.847770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6js8k\""
Jul 28 17:11:23.855411 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:23.855392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"
Jul 28 17:11:23.979045 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:23.979021    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9"]
Jul 28 17:11:24.682930 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:24.682890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9" event={"ID":"7b69a45c-150e-4efc-b0c3-2f32899dd00a","Type":"ContainerStarted","Data":"3d31f6305bb123c80fe4617c6ef50d957f83cb295d30cad96834230b7479335b"}
Jul 28 17:11:25.687764 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:25.687739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9" event={"ID":"7b69a45c-150e-4efc-b0c3-2f32899dd00a","Type":"ContainerStarted","Data":"023aa5868ca0d9a9bbf88815669185ef1e3fe07202da453d007366b33872d5fd"}
Jul 28 17:11:25.688157 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:25.687789    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9" event={"ID":"7b69a45c-150e-4efc-b0c3-2f32899dd00a","Type":"ContainerStarted","Data":"295f361464e805da93c107fa454cfce789a6dcdf823d03379c4622d1f43efadf"}
Jul 28 17:11:25.706245 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:25.706191    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-ssvr9" podStartSLOduration=17.145174292 podStartE2EDuration="18.706174463s" totalImagesPullingTime="1.561000171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:25.705046895 +0000 UTC m=+73.934246005" watchObservedRunningTime="2026-07-28 17:11:25.706174463 +0000 UTC m=+73.935373560"
Jul 28 17:11:31.314661 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:31.314626    2579 scope.go:117] "RemoveContainer" containerID="d2327ce288e67f41c4e91c54a1a5eb5ec650688de0b3489515dd89e68d1f31fb"
Jul 28 17:11:31.398646 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:31.398615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:31.398851 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:31.398756    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:31.398851 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:31.398773    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:31.398851 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:31.398816    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls podName:b88bc623-3928-4437-9acc-a10ecec23114 nodeName:}" failed. No retries permitted until 2026-07-28 17:11:47.398797769 +0000 UTC m=+95.627996864 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fg6ls" (UID: "b88bc623-3928-4437-9acc-a10ecec23114") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:11:31.399333 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:31.399306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b88bc623-3928-4437-9acc-a10ecec23114-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:31.706428 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:31.706349    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:11:31.706597 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:31.706431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt" event={"ID":"72263172-706a-41e4-89ab-b2c5a52d86ff","Type":"ContainerStarted","Data":"4339e89a3b8bf6e2f96c83b9bf6278f9b4d1a93f1a6c18baf22e476200c43938"}
Jul 28 17:11:31.706752 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:31.706733    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:31.728466 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:31.728418    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt" podStartSLOduration=18.421988608 podStartE2EDuration="23.728406109s" totalImagesPullingTime="5.306417501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:31.726791743 +0000 UTC m=+79.955990858" watchObservedRunningTime="2026-07-28 17:11:31.728406109 +0000 UTC m=+79.957605205"
Jul 28 17:11:32.066105 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:32.066064    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-2gspt"
Jul 28 17:11:39.874165 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:39.873977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:39.874165 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:39.874057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:39.874165 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:39.874100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:39.874780 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:39.874749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e77175ad-f991-48f4-bfa2-b35bdbcb138e-service-ca-bundle\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:39.876596 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:39.876564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e77175ad-f991-48f4-bfa2-b35bdbcb138e-metrics-certs\") pod \"router-default-5dc8cf7cff-gn2tw\" (UID: \"e77175ad-f991-48f4-bfa2-b35bdbcb138e\") " pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:39.876706 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:39.876564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1087a8f2-182c-40ca-95f2-9f81ce0e4bec-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-xrstz\" (UID: \"1087a8f2-182c-40ca-95f2-9f81ce0e4bec\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:40.142919 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:40.142840    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-cw6ln\""
Jul 28 17:11:40.150364 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:40.150342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"
Jul 28 17:11:40.150595 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:40.150581    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rvhbw\""
Jul 28 17:11:40.158461 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:40.158436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:40.304064 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:40.304036    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz"]
Jul 28 17:11:40.307146 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:40.307119    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1087a8f2_182c_40ca_95f2_9f81ce0e4bec.slice/crio-83041766b55a417427577fda55f382e08759604f6bbf2ddf3d8ba8fe8d369485 WatchSource:0}: Error finding container 83041766b55a417427577fda55f382e08759604f6bbf2ddf3d8ba8fe8d369485: Status 404 returned error can't find the container with id 83041766b55a417427577fda55f382e08759604f6bbf2ddf3d8ba8fe8d369485
Jul 28 17:11:40.327613 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:40.327584    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5dc8cf7cff-gn2tw"]
Jul 28 17:11:40.330439 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:40.330413    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77175ad_f991_48f4_bfa2_b35bdbcb138e.slice/crio-b2d0b6c6d45314b44b529e1698d9850e29ec74fc6c262fc765f2bd3794f4124d WatchSource:0}: Error finding container b2d0b6c6d45314b44b529e1698d9850e29ec74fc6c262fc765f2bd3794f4124d: Status 404 returned error can't find the container with id b2d0b6c6d45314b44b529e1698d9850e29ec74fc6c262fc765f2bd3794f4124d
Jul 28 17:11:40.735983 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:40.735930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz" event={"ID":"1087a8f2-182c-40ca-95f2-9f81ce0e4bec","Type":"ContainerStarted","Data":"83041766b55a417427577fda55f382e08759604f6bbf2ddf3d8ba8fe8d369485"}
Jul 28 17:11:40.742836 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:40.742799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw" event={"ID":"e77175ad-f991-48f4-bfa2-b35bdbcb138e","Type":"ContainerStarted","Data":"e0d5e09542979539a5bd15d6a85c65b02a0854a40a9f9836af535e4aabb4e7b3"}
Jul 28 17:11:40.742993 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:40.742888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw" event={"ID":"e77175ad-f991-48f4-bfa2-b35bdbcb138e","Type":"ContainerStarted","Data":"b2d0b6c6d45314b44b529e1698d9850e29ec74fc6c262fc765f2bd3794f4124d"}
Jul 28 17:11:40.770566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:40.770518    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw" podStartSLOduration=32.770502543 podStartE2EDuration="32.770502543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:40.767731228 +0000 UTC m=+88.996930348" watchObservedRunningTime="2026-07-28 17:11:40.770502543 +0000 UTC m=+88.999701640"
Jul 28 17:11:41.158976 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:41.158932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:41.161818 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:41.161792    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:41.746940 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:41.746912    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:41.748315 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:41.748285    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5dc8cf7cff-gn2tw"
Jul 28 17:11:42.751543 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:42.751501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz" event={"ID":"1087a8f2-182c-40ca-95f2-9f81ce0e4bec","Type":"ContainerStarted","Data":"3f86ee2318c07f8e95ed28b07051d4252d06fbf5b7b4e083a3fa3bab8658720b"}
Jul 28 17:11:42.770894 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:42.770848    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-xrstz" podStartSLOduration=34.252661315 podStartE2EDuration="35.770832494s" totalImagesPullingTime="1.518171179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:42.770751349 +0000 UTC m=+90.999950442" watchObservedRunningTime="2026-07-28 17:11:42.770832494 +0000 UTC m=+91.000031627"
Jul 28 17:11:47.434380 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:47.434323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:47.436635 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:47.436602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b88bc623-3928-4437-9acc-a10ecec23114-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fg6ls\" (UID: \"b88bc623-3928-4437-9acc-a10ecec23114\") " pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:47.632059 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:47.632026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fg6ls"
Jul 28 17:11:47.761039 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:47.761014    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fg6ls"]
Jul 28 17:11:47.763209 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:47.763185    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88bc623_3928_4437_9acc_a10ecec23114.slice/crio-6583082ecc7885d5deece2abb461bb34dbfa94c86da1e5ca247ede1f65ade0fe WatchSource:0}: Error finding container 6583082ecc7885d5deece2abb461bb34dbfa94c86da1e5ca247ede1f65ade0fe: Status 404 returned error can't find the container with id 6583082ecc7885d5deece2abb461bb34dbfa94c86da1e5ca247ede1f65ade0fe
Jul 28 17:11:47.768796 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:47.768770    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fg6ls" event={"ID":"b88bc623-3928-4437-9acc-a10ecec23114","Type":"ContainerStarted","Data":"6583082ecc7885d5deece2abb461bb34dbfa94c86da1e5ca247ede1f65ade0fe"}
Jul 28 17:11:48.713846 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.713560    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"]
Jul 28 17:11:48.716628 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.716595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.720516 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.720493    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 17:11:48.721815 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.721793    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4drw2\""
Jul 28 17:11:48.722073 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.722042    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 17:11:48.722191 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.722063    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:11:48.729845 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.729821    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"]
Jul 28 17:11:48.745841 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.745811    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0be57644-399e-4f8e-9a93-f0d2105f4e8f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.745955 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.745898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0be57644-399e-4f8e-9a93-f0d2105f4e8f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.745955 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.745931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7nq2\" (UniqueName: \"kubernetes.io/projected/0be57644-399e-4f8e-9a93-f0d2105f4e8f-kube-api-access-t7nq2\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.746041 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.745960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0be57644-399e-4f8e-9a93-f0d2105f4e8f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.846485 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.846448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0be57644-399e-4f8e-9a93-f0d2105f4e8f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.846672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.846499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7nq2\" (UniqueName: \"kubernetes.io/projected/0be57644-399e-4f8e-9a93-f0d2105f4e8f-kube-api-access-t7nq2\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.846672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.846532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0be57644-399e-4f8e-9a93-f0d2105f4e8f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.846672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.846615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0be57644-399e-4f8e-9a93-f0d2105f4e8f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.846827 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:48.846757    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 17:11:48.846827 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:48.846823    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0be57644-399e-4f8e-9a93-f0d2105f4e8f-prometheus-operator-tls podName:0be57644-399e-4f8e-9a93-f0d2105f4e8f nodeName:}" failed. No retries permitted until 2026-07-28 17:11:49.346806167 +0000 UTC m=+97.576005243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/0be57644-399e-4f8e-9a93-f0d2105f4e8f-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-mhmsz" (UID: "0be57644-399e-4f8e-9a93-f0d2105f4e8f") : secret "prometheus-operator-tls" not found
Jul 28 17:11:48.847424 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.847384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0be57644-399e-4f8e-9a93-f0d2105f4e8f-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.849032 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.849007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0be57644-399e-4f8e-9a93-f0d2105f4e8f-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:48.863406 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:48.863373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7nq2\" (UniqueName: \"kubernetes.io/projected/0be57644-399e-4f8e-9a93-f0d2105f4e8f-kube-api-access-t7nq2\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:49.351608 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:49.351568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0be57644-399e-4f8e-9a93-f0d2105f4e8f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:49.354367 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:49.354337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0be57644-399e-4f8e-9a93-f0d2105f4e8f-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-mhmsz\" (UID: \"0be57644-399e-4f8e-9a93-f0d2105f4e8f\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:49.630989 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:49.630914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"
Jul 28 17:11:49.664956 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:49.664790    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qr9ct"
Jul 28 17:11:49.775987 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:49.775956    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-mhmsz"]
Jul 28 17:11:49.778951 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:49.778912    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be57644_399e_4f8e_9a93_f0d2105f4e8f.slice/crio-7c58cf4532a5bd5c910b9977d15bee5b07a66266b43af5cab24991e2337b34fa WatchSource:0}: Error finding container 7c58cf4532a5bd5c910b9977d15bee5b07a66266b43af5cab24991e2337b34fa: Status 404 returned error can't find the container with id 7c58cf4532a5bd5c910b9977d15bee5b07a66266b43af5cab24991e2337b34fa
Jul 28 17:11:50.662440 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:50.662358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:11:50.662612 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:50.662440    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:11:50.665037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:50.665010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1338c56-b7f9-4bec-b8a7-8dc14677efe3-metrics-tls\") pod \"dns-default-clksn\" (UID: \"f1338c56-b7f9-4bec-b8a7-8dc14677efe3\") " pod="openshift-dns/dns-default-clksn"
Jul 28 17:11:50.665185 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:50.665129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"image-registry-5ffb6d446d-nb8db\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") " pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:11:50.782596 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:50.782558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz" event={"ID":"0be57644-399e-4f8e-9a93-f0d2105f4e8f","Type":"ContainerStarted","Data":"7c58cf4532a5bd5c910b9977d15bee5b07a66266b43af5cab24991e2337b34fa"}
Jul 28 17:11:50.871147 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:50.871114    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xks97\""
Jul 28 17:11:50.878621 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:50.878593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:11:50.892485 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:50.892466    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kslsq\""
Jul 28 17:11:50.899878 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:50.899851    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-clksn"
Jul 28 17:11:52.789952 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:52.789921    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-clksn"]
Jul 28 17:11:52.806579 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:52.806551    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5ffb6d446d-nb8db"]
Jul 28 17:11:53.087789 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:53.087759    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1338c56_b7f9_4bec_b8a7_8dc14677efe3.slice/crio-1ff3c7e2f254b8924675babf41046293a6d2cf981482441acee6729b2cf504e0 WatchSource:0}: Error finding container 1ff3c7e2f254b8924675babf41046293a6d2cf981482441acee6729b2cf504e0: Status 404 returned error can't find the container with id 1ff3c7e2f254b8924675babf41046293a6d2cf981482441acee6729b2cf504e0
Jul 28 17:11:53.088182 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:53.088153    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7068fb4a_e8f2_4262_9873_da9abde135ac.slice/crio-9e67943aebf764c03f1effafbc3d0c795976a8b7ce07cefc756e598b089068cc WatchSource:0}: Error finding container 9e67943aebf764c03f1effafbc3d0c795976a8b7ce07cefc756e598b089068cc: Status 404 returned error can't find the container with id 9e67943aebf764c03f1effafbc3d0c795976a8b7ce07cefc756e598b089068cc
Jul 28 17:11:53.796004 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:53.795915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz" event={"ID":"0be57644-399e-4f8e-9a93-f0d2105f4e8f","Type":"ContainerStarted","Data":"c39a02a4298906512795993e9d26588b60fc14588e2dbccd04bd7145100698c9"}
Jul 28 17:11:53.796004 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:53.795981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz" event={"ID":"0be57644-399e-4f8e-9a93-f0d2105f4e8f","Type":"ContainerStarted","Data":"36c4059950e1fe4ebfd4e84e3b4c75beadabab8eee39a433b2112461a1289358"}
Jul 28 17:11:53.802970 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:53.802930    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clksn" event={"ID":"f1338c56-b7f9-4bec-b8a7-8dc14677efe3","Type":"ContainerStarted","Data":"1ff3c7e2f254b8924675babf41046293a6d2cf981482441acee6729b2cf504e0"}
Jul 28 17:11:53.805615 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:53.805587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fg6ls" event={"ID":"b88bc623-3928-4437-9acc-a10ecec23114","Type":"ContainerStarted","Data":"b8c2bdd7eb07a1a5a69a4f06147bfb3a3aa028127902a3e9746d5d0cc822afc8"}
Jul 28 17:11:53.807337 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:53.807308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db" event={"ID":"7068fb4a-e8f2-4262-9873-da9abde135ac","Type":"ContainerStarted","Data":"163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976"}
Jul 28 17:11:53.807470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:53.807355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db" event={"ID":"7068fb4a-e8f2-4262-9873-da9abde135ac","Type":"ContainerStarted","Data":"9e67943aebf764c03f1effafbc3d0c795976a8b7ce07cefc756e598b089068cc"}
Jul 28 17:11:53.807470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:53.807453    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:11:53.824740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:53.824692    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-mhmsz" podStartSLOduration=2.280477527 podStartE2EDuration="5.82467242s" totalImagesPullingTime="3.544194893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:53.823507035 +0000 UTC m=+102.052706130" watchObservedRunningTime="2026-07-28 17:11:53.82467242 +0000 UTC m=+102.053871513"
Jul 28 17:11:53.862994 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:53.862939    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db" podStartSLOduration=101.862920215 podStartE2EDuration="1m41.862920215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:53.861811378 +0000 UTC m=+102.091010474" watchObservedRunningTime="2026-07-28 17:11:53.862920215 +0000 UTC m=+102.092119312"
Jul 28 17:11:55.824047 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:55.824004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clksn" event={"ID":"f1338c56-b7f9-4bec-b8a7-8dc14677efe3","Type":"ContainerStarted","Data":"c177f15ba97188cfe20e44ee8ff7bd8037b7e9565f64c8209a7fc504903f7fc7"}
Jul 28 17:11:56.117183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.117098    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vvzl4"]
Jul 28 17:11:56.135179 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.135148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.139901 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.139582    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:11:56.139901 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.139584    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:11:56.139901 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.139869    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:11:56.139901 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.139887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pzgw5\""
Jul 28 17:11:56.221201 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.221160    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-tls\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.221201 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.221203    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45737a07-b27b-4d1f-9603-93e426b9796b-root\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.221448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.221261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-wtmp\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.221448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.221331    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45737a07-b27b-4d1f-9603-93e426b9796b-metrics-client-ca\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.221448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.221386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7n2\" (UniqueName: \"kubernetes.io/projected/45737a07-b27b-4d1f-9603-93e426b9796b-kube-api-access-fx7n2\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.221448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.221443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-textfile\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.221639 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.221497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.221639 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.221580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.221639 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.221615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45737a07-b27b-4d1f-9603-93e426b9796b-sys\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.322618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322315    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-wtmp\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.322618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45737a07-b27b-4d1f-9603-93e426b9796b-metrics-client-ca\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.322618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7n2\" (UniqueName: \"kubernetes.io/projected/45737a07-b27b-4d1f-9603-93e426b9796b-kube-api-access-fx7n2\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.322618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322418    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-textfile\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.322618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322465    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.322618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322511    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.322618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45737a07-b27b-4d1f-9603-93e426b9796b-sys\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.322618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-tls\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.322618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45737a07-b27b-4d1f-9603-93e426b9796b-root\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.323137 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322659    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/45737a07-b27b-4d1f-9603-93e426b9796b-sys\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.323137 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322729    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/45737a07-b27b-4d1f-9603-93e426b9796b-root\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.323137 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322755    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-textfile\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.323137 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:56.322765    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:11:56.323137 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:11:56.322822    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-tls podName:45737a07-b27b-4d1f-9603-93e426b9796b nodeName:}" failed. No retries permitted until 2026-07-28 17:11:56.822803447 +0000 UTC m=+105.052002525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-tls") pod "node-exporter-vvzl4" (UID: "45737a07-b27b-4d1f-9603-93e426b9796b") : secret "node-exporter-tls" not found
Jul 28 17:11:56.323137 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.322869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-wtmp\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.323137 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.323036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/45737a07-b27b-4d1f-9603-93e426b9796b-metrics-client-ca\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.323137 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.323123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-accelerators-collector-config\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.325166 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.325135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.331547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.331525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7n2\" (UniqueName: \"kubernetes.io/projected/45737a07-b27b-4d1f-9603-93e426b9796b-kube-api-access-fx7n2\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.828018 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.827985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-tls\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:56.830846 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:56.830821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/45737a07-b27b-4d1f-9603-93e426b9796b-node-exporter-tls\") pod \"node-exporter-vvzl4\" (UID: \"45737a07-b27b-4d1f-9603-93e426b9796b\") " pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:57.047668 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.047625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vvzl4"
Jul 28 17:11:57.061762 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:57.061733    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45737a07_b27b_4d1f_9603_93e426b9796b.slice/crio-a7e3647f9279e09a436f8726c912056a1f525b1f8d4dc4b5ddfb28178f71175e WatchSource:0}: Error finding container a7e3647f9279e09a436f8726c912056a1f525b1f8d4dc4b5ddfb28178f71175e: Status 404 returned error can't find the container with id a7e3647f9279e09a436f8726c912056a1f525b1f8d4dc4b5ddfb28178f71175e
Jul 28 17:11:57.198329 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.198296    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:11:57.201723 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.201697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.205378 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.205346    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:11:57.205499 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.205429    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:11:57.205573 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.205429    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:11:57.205800 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.205776    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:11:57.205800 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.205792    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:11:57.205944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.205828    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:11:57.205944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.205835    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-qj8hg\""
Jul 28 17:11:57.205944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.205785    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:11:57.205944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.205915    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:11:57.206213 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.206195    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:11:57.215913 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.215894    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:11:57.332556 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9sd\" (UniqueName: \"kubernetes.io/projected/642803fa-d182-445e-a4da-f386cb75b3b0-kube-api-access-9r9sd\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.332556 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332524    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.332556 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/642803fa-d182-445e-a4da-f386cb75b3b0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.332756 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-web-config\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.332756 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.332756 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.332756 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332731    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/642803fa-d182-445e-a4da-f386cb75b3b0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.332902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332789    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/642803fa-d182-445e-a4da-f386cb75b3b0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.332902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/642803fa-d182-445e-a4da-f386cb75b3b0-config-out\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.332902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/642803fa-d182-445e-a4da-f386cb75b3b0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.332902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.333098 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.333098 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.332932    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-config-volume\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.433665 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.433622    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/642803fa-d182-445e-a4da-f386cb75b3b0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.433848 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.433676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.433848 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.433728    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.433848 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.433776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-config-volume\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.433999 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.433854    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9sd\" (UniqueName: \"kubernetes.io/projected/642803fa-d182-445e-a4da-f386cb75b3b0-kube-api-access-9r9sd\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.433999 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.433883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.433999 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.433917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/642803fa-d182-445e-a4da-f386cb75b3b0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.433999 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.433944    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-web-config\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.433999 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.433975    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.434254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.434022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.434254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.434053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/642803fa-d182-445e-a4da-f386cb75b3b0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.434254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.434103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/642803fa-d182-445e-a4da-f386cb75b3b0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.434254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.434135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/642803fa-d182-445e-a4da-f386cb75b3b0-config-out\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.435381 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.435141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/642803fa-d182-445e-a4da-f386cb75b3b0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.435520 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.435487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/642803fa-d182-445e-a4da-f386cb75b3b0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.435722 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.435696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/642803fa-d182-445e-a4da-f386cb75b3b0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.436717 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.436692    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/642803fa-d182-445e-a4da-f386cb75b3b0-config-out\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.436861 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.436840    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.437828 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.437802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.438054 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.438030    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-web-config\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.438529 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.438503    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.438632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.438565    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-config-volume\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.438632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.438572    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/642803fa-d182-445e-a4da-f386cb75b3b0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.439258 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.439231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.439365 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.439348    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/642803fa-d182-445e-a4da-f386cb75b3b0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.446818 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.446796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9sd\" (UniqueName: \"kubernetes.io/projected/642803fa-d182-445e-a4da-f386cb75b3b0-kube-api-access-9r9sd\") pod \"alertmanager-main-0\" (UID: \"642803fa-d182-445e-a4da-f386cb75b3b0\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.514280 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.514252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:11:57.678758 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.678466    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:11:57.683112 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:11:57.682832    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642803fa_d182_445e_a4da_f386cb75b3b0.slice/crio-be21ef01e07eec6284dec04d4a8863d76172f8370df970f97b0c6b76e5bf6a2c WatchSource:0}: Error finding container be21ef01e07eec6284dec04d4a8863d76172f8370df970f97b0c6b76e5bf6a2c: Status 404 returned error can't find the container with id be21ef01e07eec6284dec04d4a8863d76172f8370df970f97b0c6b76e5bf6a2c
Jul 28 17:11:57.833642 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.833576    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"642803fa-d182-445e-a4da-f386cb75b3b0","Type":"ContainerStarted","Data":"be21ef01e07eec6284dec04d4a8863d76172f8370df970f97b0c6b76e5bf6a2c"}
Jul 28 17:11:57.836806 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.836739    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fg6ls" event={"ID":"b88bc623-3928-4437-9acc-a10ecec23114","Type":"ContainerStarted","Data":"736a80450a2ddcbe0e7a629da91cc6c392cd1c646b772c06112d78e83b1f5a8c"}
Jul 28 17:11:57.841246 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.841021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clksn" event={"ID":"f1338c56-b7f9-4bec-b8a7-8dc14677efe3","Type":"ContainerStarted","Data":"526278d04880ebe59f0a7bf51828d267c8fa5131a2bb32a4ebf6a53040aafabd"}
Jul 28 17:11:57.842119 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.842044    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-clksn"
Jul 28 17:11:57.844487 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.844391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvzl4" event={"ID":"45737a07-b27b-4d1f-9603-93e426b9796b","Type":"ContainerStarted","Data":"a7e3647f9279e09a436f8726c912056a1f525b1f8d4dc4b5ddfb28178f71175e"}
Jul 28 17:11:57.864396 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:57.864306    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-clksn" podStartSLOduration=70.007620818 podStartE2EDuration="1m11.864285305s" totalImagesPullingTime="1.856664487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:57.86263631 +0000 UTC m=+106.091835405" watchObservedRunningTime="2026-07-28 17:11:57.864285305 +0000 UTC m=+106.093484402"
Jul 28 17:11:58.851486 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:58.851451    2579 generic.go:356] "Generic (PLEG): container finished" podID="45737a07-b27b-4d1f-9603-93e426b9796b" containerID="32a8cac24663b40806102763f74e5cbe15cbf3e2b7a81160251c5d5c5baec9e0" exitCode=0
Jul 28 17:11:58.851928 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:58.851541    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvzl4" event={"ID":"45737a07-b27b-4d1f-9603-93e426b9796b","Type":"ContainerDied","Data":"32a8cac24663b40806102763f74e5cbe15cbf3e2b7a81160251c5d5c5baec9e0"}
Jul 28 17:11:59.170394 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.169335    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67bb788c5-nt92b"]
Jul 28 17:11:59.173035 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.173005    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.175812 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.175786    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 28 17:11:59.175955 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.175831    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 28 17:11:59.175955 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.175786    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 28 17:11:59.176262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.176236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fzzmr\""
Jul 28 17:11:59.176262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.176254    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 28 17:11:59.176442 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.176345    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 28 17:11:59.176512 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.176452    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3ami4n4rn9s9c\""
Jul 28 17:11:59.184593 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.184528    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67bb788c5-nt92b"]
Jul 28 17:11:59.253672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.253233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.253672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.253280    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.253672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.253301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.253672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.253328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/491df11b-d7b8-406e-a6ed-8a338b3c626c-metrics-client-ca\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.253672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.253428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-grpc-tls\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.253672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.253460    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-tls\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.253672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.253514    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.253672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.253539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5k6j\" (UniqueName: \"kubernetes.io/projected/491df11b-d7b8-406e-a6ed-8a338b3c626c-kube-api-access-p5k6j\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.354771 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.354734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-tls\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.354939 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.354779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.354939 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.354903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5k6j\" (UniqueName: \"kubernetes.io/projected/491df11b-d7b8-406e-a6ed-8a338b3c626c-kube-api-access-p5k6j\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.355048 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.354965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.355048 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.354994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.355048 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.355012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.355048 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.355041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/491df11b-d7b8-406e-a6ed-8a338b3c626c-metrics-client-ca\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.355290 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.355218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-grpc-tls\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.356218 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.356163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/491df11b-d7b8-406e-a6ed-8a338b3c626c-metrics-client-ca\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.357796 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.357775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-tls\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.357906 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.357859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.358175 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.358141    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-grpc-tls\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.358516 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.358494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.358756 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.358724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.358895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.358877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/491df11b-d7b8-406e-a6ed-8a338b3c626c-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.363211 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.363189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5k6j\" (UniqueName: \"kubernetes.io/projected/491df11b-d7b8-406e-a6ed-8a338b3c626c-kube-api-access-p5k6j\") pod \"thanos-querier-67bb788c5-nt92b\" (UID: \"491df11b-d7b8-406e-a6ed-8a338b3c626c\") " pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.486143 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.486045    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:11:59.858949 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.858908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvzl4" event={"ID":"45737a07-b27b-4d1f-9603-93e426b9796b","Type":"ContainerStarted","Data":"adc9e273c3d69602960e409fdf31bc037fc9ce26801a587583ef2011ee382c38"}
Jul 28 17:11:59.859440 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.858981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vvzl4" event={"ID":"45737a07-b27b-4d1f-9603-93e426b9796b","Type":"ContainerStarted","Data":"9342b1a18caca89fc61140a5afc939b0b36faaccd767fc51d3cfd0e071198b3b"}
Jul 28 17:11:59.879600 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:11:59.879537    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vvzl4" podStartSLOduration=2.722265286 podStartE2EDuration="3.879506267s" totalImagesPullingTime="1.083243494s" totalInitContainerRuntime="73.997487ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:11:59.879276995 +0000 UTC m=+108.108476112" watchObservedRunningTime="2026-07-28 17:11:59.879506267 +0000 UTC m=+108.108705365"
Jul 28 17:12:00.861822 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:00.861782    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"]
Jul 28 17:12:00.885509 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:00.885470    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"]
Jul 28 17:12:00.885711 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:00.885585    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:00.888156 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:00.888131    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 17:12:00.888267 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:00.888201    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-447nc\""
Jul 28 17:12:00.972997 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:00.972939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc5r8\" (UniqueName: \"kubernetes.io/projected/d59926dd-c34f-4486-bed5-853d8eb97e68-kube-api-access-bc5r8\") pod \"monitoring-plugin-68b7fc9574-pq9fh\" (UID: \"d59926dd-c34f-4486-bed5-853d8eb97e68\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:00.973185 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:00.973113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d59926dd-c34f-4486-bed5-853d8eb97e68-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pq9fh\" (UID: \"d59926dd-c34f-4486-bed5-853d8eb97e68\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:01.073923 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.073886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc5r8\" (UniqueName: \"kubernetes.io/projected/d59926dd-c34f-4486-bed5-853d8eb97e68-kube-api-access-bc5r8\") pod \"monitoring-plugin-68b7fc9574-pq9fh\" (UID: \"d59926dd-c34f-4486-bed5-853d8eb97e68\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:01.074106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.073971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d59926dd-c34f-4486-bed5-853d8eb97e68-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pq9fh\" (UID: \"d59926dd-c34f-4486-bed5-853d8eb97e68\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:01.074167 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:12:01.074101    2579 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 17:12:01.074167 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:12:01.074161    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59926dd-c34f-4486-bed5-853d8eb97e68-monitoring-plugin-cert podName:d59926dd-c34f-4486-bed5-853d8eb97e68 nodeName:}" failed. No retries permitted until 2026-07-28 17:12:01.574145612 +0000 UTC m=+109.803344687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d59926dd-c34f-4486-bed5-853d8eb97e68-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-pq9fh" (UID: "d59926dd-c34f-4486-bed5-853d8eb97e68") : secret "monitoring-plugin-cert" not found
Jul 28 17:12:01.085188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.085155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc5r8\" (UniqueName: \"kubernetes.io/projected/d59926dd-c34f-4486-bed5-853d8eb97e68-kube-api-access-bc5r8\") pod \"monitoring-plugin-68b7fc9574-pq9fh\" (UID: \"d59926dd-c34f-4486-bed5-853d8eb97e68\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:01.367477 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.367444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67bb788c5-nt92b"]
Jul 28 17:12:01.371231 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:12:01.371207    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491df11b_d7b8_406e_a6ed_8a338b3c626c.slice/crio-07f6dacc45768d2e708b4b421f68499d351bde661945ff16892a96e44df97066 WatchSource:0}: Error finding container 07f6dacc45768d2e708b4b421f68499d351bde661945ff16892a96e44df97066: Status 404 returned error can't find the container with id 07f6dacc45768d2e708b4b421f68499d351bde661945ff16892a96e44df97066
Jul 28 17:12:01.578131 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.578098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d59926dd-c34f-4486-bed5-853d8eb97e68-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pq9fh\" (UID: \"d59926dd-c34f-4486-bed5-853d8eb97e68\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:01.580459 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.580433    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d59926dd-c34f-4486-bed5-853d8eb97e68-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-pq9fh\" (UID: \"d59926dd-c34f-4486-bed5-853d8eb97e68\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:01.795988 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.795956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:01.869546 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.869511    2579 generic.go:356] "Generic (PLEG): container finished" podID="642803fa-d182-445e-a4da-f386cb75b3b0" containerID="b288388e203b3540557809748b5144dbcb4b313437f7431c03a4f402d92cae0b" exitCode=0
Jul 28 17:12:01.869910 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.869590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"642803fa-d182-445e-a4da-f386cb75b3b0","Type":"ContainerDied","Data":"b288388e203b3540557809748b5144dbcb4b313437f7431c03a4f402d92cae0b"}
Jul 28 17:12:01.871849 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.871813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fg6ls" event={"ID":"b88bc623-3928-4437-9acc-a10ecec23114","Type":"ContainerStarted","Data":"68115e7753603baf51c804fbc4429a59bc423e8460ea557d34e8949b90b9a257"}
Jul 28 17:12:01.873787 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.873670    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b" event={"ID":"491df11b-d7b8-406e-a6ed-8a338b3c626c","Type":"ContainerStarted","Data":"07f6dacc45768d2e708b4b421f68499d351bde661945ff16892a96e44df97066"}
Jul 28 17:12:01.923121 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.923073    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"]
Jul 28 17:12:01.925705 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:12:01.925679    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59926dd_c34f_4486_bed5_853d8eb97e68.slice/crio-fd159aff7a3ed9c32b025aad2bd7e7fae044dca717d172c436153441f47fa16f WatchSource:0}: Error finding container fd159aff7a3ed9c32b025aad2bd7e7fae044dca717d172c436153441f47fa16f: Status 404 returned error can't find the container with id fd159aff7a3ed9c32b025aad2bd7e7fae044dca717d172c436153441f47fa16f
Jul 28 17:12:01.930646 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:01.930601    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fg6ls" podStartSLOduration=33.647001618 podStartE2EDuration="46.930586833s" totalImagesPullingTime="13.283585215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:11:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:01.930247725 +0000 UTC m=+110.159446839" watchObservedRunningTime="2026-07-28 17:12:01.930586833 +0000 UTC m=+110.159785928"
Jul 28 17:12:02.364072 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.364036    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:12:02.370030 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.368505    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.372422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.371702    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 17:12:02.372422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.371771    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fbqxb\""
Jul 28 17:12:02.372422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.372014    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1mr7rasj5p3fq\""
Jul 28 17:12:02.372422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.372052    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:12:02.372422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.372227    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 17:12:02.372422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.372256    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 17:12:02.372422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.372404    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 17:12:02.372956 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.372781    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 17:12:02.373050 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.372988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 17:12:02.373242 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.373221    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 17:12:02.375248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.373352    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 17:12:02.375248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.373414    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 17:12:02.375248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.373552    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 17:12:02.375248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.373639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 17:12:02.375248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.373786    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 17:12:02.375248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.374110    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 17:12:02.377259 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.377237    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 17:12:02.383247 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.380789    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488545    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-config\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488679    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72z8r\" (UniqueName: \"kubernetes.io/projected/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-kube-api-access-72z8r\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.489973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.490958 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.490958 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.490958 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.488980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.490958 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.489020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.589963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590054    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-config\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590453    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590530    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.591427 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.592383 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72z8r\" (UniqueName: \"kubernetes.io/projected/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-kube-api-access-72z8r\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.592383 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.592383 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.590640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.595254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.592642    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.595254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.593028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.595254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.594254    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.595254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.594893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.595552 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.595480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.596188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.596160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.597120 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.596763    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.597120 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.596919    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.601271 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.601247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.604354 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.603961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.604354 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.603998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-web-config\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.605629 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.604838    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-config\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.605629 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.605245    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.605629 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.605328    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.605837 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.605710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.606405 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.606349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.606521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.606496    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.607262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.606722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.607262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.607104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72z8r\" (UniqueName: \"kubernetes.io/projected/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-kube-api-access-72z8r\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.607262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.607051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/73b2cd2f-d3ba-44e1-af6d-98af9c5548e4-config-out\") pod \"prometheus-k8s-0\" (UID: \"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.686706 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.686509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:02.883342 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.883306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh" event={"ID":"d59926dd-c34f-4486-bed5-853d8eb97e68","Type":"ContainerStarted","Data":"fd159aff7a3ed9c32b025aad2bd7e7fae044dca717d172c436153441f47fa16f"}
Jul 28 17:12:02.899127 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:12:02.898367    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b2cd2f_d3ba_44e1_af6d_98af9c5548e4.slice/crio-2f139cfdfbfcea6d2f4843989debb08ef1a0b4ea48b3cb105d2bbab7fdb511b2 WatchSource:0}: Error finding container 2f139cfdfbfcea6d2f4843989debb08ef1a0b4ea48b3cb105d2bbab7fdb511b2: Status 404 returned error can't find the container with id 2f139cfdfbfcea6d2f4843989debb08ef1a0b4ea48b3cb105d2bbab7fdb511b2
Jul 28 17:12:02.905724 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:02.905696    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:12:03.889809 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:03.889701    2579 generic.go:356] "Generic (PLEG): container finished" podID="73b2cd2f-d3ba-44e1-af6d-98af9c5548e4" containerID="f079b8df5d92e43b93fe32d3d82c5ed76adb4d041c1053e04c4e6e1bc7bde31c" exitCode=0
Jul 28 17:12:03.889809 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:03.889794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4","Type":"ContainerDied","Data":"f079b8df5d92e43b93fe32d3d82c5ed76adb4d041c1053e04c4e6e1bc7bde31c"}
Jul 28 17:12:03.890328 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:03.889860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4","Type":"ContainerStarted","Data":"2f139cfdfbfcea6d2f4843989debb08ef1a0b4ea48b3cb105d2bbab7fdb511b2"}
Jul 28 17:12:04.628240 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.628210    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ffb6d446d-nb8db"]
Jul 28 17:12:04.896832 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.896797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b" event={"ID":"491df11b-d7b8-406e-a6ed-8a338b3c626c","Type":"ContainerStarted","Data":"cde6aba461143b9ef59f61bea68939728073c507239e5f78b9b1be20837bee96"}
Jul 28 17:12:04.897244 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.896883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b" event={"ID":"491df11b-d7b8-406e-a6ed-8a338b3c626c","Type":"ContainerStarted","Data":"9b15ccda4744b93017cf39a8c5dcf8601283bbc77ff0a18f67bf63457585ff53"}
Jul 28 17:12:04.897244 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.896927    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b" event={"ID":"491df11b-d7b8-406e-a6ed-8a338b3c626c","Type":"ContainerStarted","Data":"d0ceb5025a3b9c1dc4b1f16b819659befec7979d833ea1078e5688497d627288"}
Jul 28 17:12:04.898514 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.898484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh" event={"ID":"d59926dd-c34f-4486-bed5-853d8eb97e68","Type":"ContainerStarted","Data":"eafec9e11a50fcdbdf95a84a59a6536edbe9b35c01c1d49c41bf9bc95a5a13db"}
Jul 28 17:12:04.898721 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.898698    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:04.902108 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.902063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"642803fa-d182-445e-a4da-f386cb75b3b0","Type":"ContainerStarted","Data":"6bdc6e900e0fbff523d08b0c19cd851fa2371557240381b4cd45c4dadecfdecb"}
Jul 28 17:12:04.902229 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.902135    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"642803fa-d182-445e-a4da-f386cb75b3b0","Type":"ContainerStarted","Data":"5747f3b61a3158d4df2c4f74e668c6b354f27a7342d056f256b8cdda244c288c"}
Jul 28 17:12:04.902229 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.902161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"642803fa-d182-445e-a4da-f386cb75b3b0","Type":"ContainerStarted","Data":"21fee9117d3156082ce822af901c497d50fcd78d7b3f3dbe0567abd4801624fe"}
Jul 28 17:12:04.902229 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.902201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"642803fa-d182-445e-a4da-f386cb75b3b0","Type":"ContainerStarted","Data":"cbab8dd76bbc3b24c0eb1bf04e5d7ba90cf77dec37056d6003891c8fc8263d53"}
Jul 28 17:12:04.902229 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.902229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"642803fa-d182-445e-a4da-f386cb75b3b0","Type":"ContainerStarted","Data":"6ba42c0b66be70940fc88a88b35b0ced73093df15f406e4ce4ee0d253be00869"}
Jul 28 17:12:04.904904 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.904883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh"
Jul 28 17:12:04.924157 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:04.924114    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-pq9fh" podStartSLOduration=2.548686018 podStartE2EDuration="4.924097766s" totalImagesPullingTime="2.375411748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:04.921955124 +0000 UTC m=+113.151154221" watchObservedRunningTime="2026-07-28 17:12:04.924097766 +0000 UTC m=+113.153296865"
Jul 28 17:12:05.861000 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:05.860969    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-clksn"
Jul 28 17:12:05.911871 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:05.911835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b" event={"ID":"491df11b-d7b8-406e-a6ed-8a338b3c626c","Type":"ContainerStarted","Data":"9c3bf9232994124087731de4b7bf8f62b97b84d144bb63610a7b6d0e8d54974a"}
Jul 28 17:12:05.912325 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:05.911920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b" event={"ID":"491df11b-d7b8-406e-a6ed-8a338b3c626c","Type":"ContainerStarted","Data":"99d1941d8407f0408ef5f264ae5e398d060ebac6e02fb3ca3ade513cd1eda541"}
Jul 28 17:12:05.912325 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:05.911970    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b" event={"ID":"491df11b-d7b8-406e-a6ed-8a338b3c626c","Type":"ContainerStarted","Data":"43022ff40f586e3057a9be746bf1f50cb57dac398de2fa9ee2328923524050f2"}
Jul 28 17:12:05.912325 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:05.912018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:12:05.916389 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:05.916358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"642803fa-d182-445e-a4da-f386cb75b3b0","Type":"ContainerStarted","Data":"f8dc4c9efc972492efb69b74834fadb3b93b94b516579c6125ca37fa148ffa87"}
Jul 28 17:12:05.934542 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:05.934323    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b" podStartSLOduration=3.380578474 podStartE2EDuration="6.93430496s" totalImagesPullingTime="3.553726486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:05.933736174 +0000 UTC m=+114.162935271" watchObservedRunningTime="2026-07-28 17:12:05.93430496 +0000 UTC m=+114.163504056"
Jul 28 17:12:05.962790 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:05.962744    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.507927777 podStartE2EDuration="8.96272896s" totalImagesPullingTime="6.436646826s" totalInitContainerRuntime="18.154357ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:11:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:05.961244744 +0000 UTC m=+114.190443873" watchObservedRunningTime="2026-07-28 17:12:05.96272896 +0000 UTC m=+114.191928057"
Jul 28 17:12:07.932837 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:07.932804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4","Type":"ContainerStarted","Data":"b058b96b0ca075a61f45b7fe23cd9204956c75f9d2b1a1030912da2b7ab51a34"}
Jul 28 17:12:07.933303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:07.932891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4","Type":"ContainerStarted","Data":"d5a1eb5b09bfbc1cf5d038b2f9630fb78c470d66d7ddbd1cb1558f2ae1c0a5d0"}
Jul 28 17:12:07.933303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:07.932917    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4","Type":"ContainerStarted","Data":"a86ff6ba266ef4f0630b3bcd224aa51effbb99fe7b16c67936878f85b7164e4c"}
Jul 28 17:12:07.933303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:07.932941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4","Type":"ContainerStarted","Data":"5122eadcebdfb3d1452fa96636d972c85b13a7c865b63cbe4bbfd1abe5649bfa"}
Jul 28 17:12:07.933303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:07.932964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4","Type":"ContainerStarted","Data":"71237f5fd2161bb23571d4aeeb0d068127eb59517970392341859e31fc68c1c0"}
Jul 28 17:12:07.933303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:07.932991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"73b2cd2f-d3ba-44e1-af6d-98af9c5548e4","Type":"ContainerStarted","Data":"9cce2f6198dbb4905e202bc07b3d5e15714d34675154e657b466a86a51220112"}
Jul 28 17:12:07.971824 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:07.971769    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.911802372 podStartE2EDuration="5.971754133s" totalImagesPullingTime="3.034635245s" totalInitContainerRuntime="25.316516ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:07.969903169 +0000 UTC m=+116.199102266" watchObservedRunningTime="2026-07-28 17:12:07.971754133 +0000 UTC m=+116.200953228"
Jul 28 17:12:11.928654 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:11.928628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67bb788c5-nt92b"
Jul 28 17:12:12.687383 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:12.687344    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:12:14.635303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:14.635271    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:12:22.171760 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:22.171722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:12:22.174111 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:22.174069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/19846f2a-05eb-485f-8ae6-67e7090bc346-metrics-certs\") pod \"network-metrics-daemon-bx6mn\" (UID: \"19846f2a-05eb-485f-8ae6-67e7090bc346\") " pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:12:22.226469 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:22.226431    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8c6kz\""
Jul 28 17:12:22.234068 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:22.234038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bx6mn"
Jul 28 17:12:22.476689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:22.476563    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bx6mn"]
Jul 28 17:12:22.478987 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:12:22.478961    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19846f2a_05eb_485f_8ae6_67e7090bc346.slice/crio-ef62c9d5967bee285da6471c7d6e1aa3aa4eebb986b8c203599ac5b3a044e509 WatchSource:0}: Error finding container ef62c9d5967bee285da6471c7d6e1aa3aa4eebb986b8c203599ac5b3a044e509: Status 404 returned error can't find the container with id ef62c9d5967bee285da6471c7d6e1aa3aa4eebb986b8c203599ac5b3a044e509
Jul 28 17:12:22.991969 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:22.991936    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bx6mn" event={"ID":"19846f2a-05eb-485f-8ae6-67e7090bc346","Type":"ContainerStarted","Data":"ef62c9d5967bee285da6471c7d6e1aa3aa4eebb986b8c203599ac5b3a044e509"}
Jul 28 17:12:25.006941 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:25.006883    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bx6mn" event={"ID":"19846f2a-05eb-485f-8ae6-67e7090bc346","Type":"ContainerStarted","Data":"738beb06169d30f376790e1035a9c7e2a2971c2e03f50b0ade4a9eefbe6d88ab"}
Jul 28 17:12:25.007374 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:25.006965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bx6mn" event={"ID":"19846f2a-05eb-485f-8ae6-67e7090bc346","Type":"ContainerStarted","Data":"f001b3c25cf602888560d831197087ea2f94e02fd1965bc1481ce4de31b5fd4d"}
Jul 28 17:12:25.041485 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:25.041432    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bx6mn" podStartSLOduration=131.527418899 podStartE2EDuration="2m13.041417094s" totalImagesPullingTime="1.513998195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:10:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:12:25.041163743 +0000 UTC m=+133.270362840" watchObservedRunningTime="2026-07-28 17:12:25.041417094 +0000 UTC m=+133.270616190"
Jul 28 17:12:29.649582 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:29.649544    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db" podUID="7068fb4a-e8f2-4262-9873-da9abde135ac" containerName="registry" containerID="cri-o://163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976" gracePeriod=30
Jul 28 17:12:29.972301 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:29.972268    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:12:30.029035 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.029002    2579 generic.go:356] "Generic (PLEG): container finished" podID="e62fc7a6-3156-4e2d-a376-1a753ef8eb8d" containerID="882b7aa797caf81eddc90ce76ca2271c43011a996882db9756cc45a56665325b" exitCode=0
Jul 28 17:12:30.029209 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.029097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n" event={"ID":"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d","Type":"ContainerDied","Data":"882b7aa797caf81eddc90ce76ca2271c43011a996882db9756cc45a56665325b"}
Jul 28 17:12:30.029517 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.029495    2579 scope.go:117] "RemoveContainer" containerID="882b7aa797caf81eddc90ce76ca2271c43011a996882db9756cc45a56665325b"
Jul 28 17:12:30.030342 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.030322    2579 generic.go:356] "Generic (PLEG): container finished" podID="7068fb4a-e8f2-4262-9873-da9abde135ac" containerID="163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976" exitCode=0
Jul 28 17:12:30.030432 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.030418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db" event={"ID":"7068fb4a-e8f2-4262-9873-da9abde135ac","Type":"ContainerDied","Data":"163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976"}
Jul 28 17:12:30.030470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.030439    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db"
Jul 28 17:12:30.030470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.030451    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5ffb6d446d-nb8db" event={"ID":"7068fb4a-e8f2-4262-9873-da9abde135ac","Type":"ContainerDied","Data":"9e67943aebf764c03f1effafbc3d0c795976a8b7ce07cefc756e598b089068cc"}
Jul 28 17:12:30.030534 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.030469    2579 scope.go:117] "RemoveContainer" containerID="163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976"
Jul 28 17:12:30.041162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.041146    2579 scope.go:117] "RemoveContainer" containerID="163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976"
Jul 28 17:12:30.041418 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:12:30.041390    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976\": container with ID starting with 163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976 not found: ID does not exist" containerID="163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976"
Jul 28 17:12:30.041469 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.041426    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976"} err="failed to get container status \"163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976\": rpc error: code = NotFound desc = could not find container \"163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976\": container with ID starting with 163d57e6f7be1e8486890e487cb8af7b87740f45ff926d9fc9725109ddfa2976 not found: ID does not exist"
Jul 28 17:12:30.044755 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.044736    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-trusted-ca\") pod \"7068fb4a-e8f2-4262-9873-da9abde135ac\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") "
Jul 28 17:12:30.044840 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.044782    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7068fb4a-e8f2-4262-9873-da9abde135ac-ca-trust-extracted\") pod \"7068fb4a-e8f2-4262-9873-da9abde135ac\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") "
Jul 28 17:12:30.044840 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.044822    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") pod \"7068fb4a-e8f2-4262-9873-da9abde135ac\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") "
Jul 28 17:12:30.044945 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.044865    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-certificates\") pod \"7068fb4a-e8f2-4262-9873-da9abde135ac\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") "
Jul 28 17:12:30.044945 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.044932    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-bound-sa-token\") pod \"7068fb4a-e8f2-4262-9873-da9abde135ac\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") "
Jul 28 17:12:30.045528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.045115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-installation-pull-secrets\") pod \"7068fb4a-e8f2-4262-9873-da9abde135ac\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") "
Jul 28 17:12:30.045528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.045155    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-image-registry-private-configuration\") pod \"7068fb4a-e8f2-4262-9873-da9abde135ac\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") "
Jul 28 17:12:30.045528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.045186    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t56kz\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-kube-api-access-t56kz\") pod \"7068fb4a-e8f2-4262-9873-da9abde135ac\" (UID: \"7068fb4a-e8f2-4262-9873-da9abde135ac\") "
Jul 28 17:12:30.045528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.045412    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7068fb4a-e8f2-4262-9873-da9abde135ac" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:30.045528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.045501    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-certificates\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:12:30.046670 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.045554    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7068fb4a-e8f2-4262-9873-da9abde135ac" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:12:30.048214 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.048160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-kube-api-access-t56kz" (OuterVolumeSpecName: "kube-api-access-t56kz") pod "7068fb4a-e8f2-4262-9873-da9abde135ac" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac"). InnerVolumeSpecName "kube-api-access-t56kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:12:30.048535 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.048496    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7068fb4a-e8f2-4262-9873-da9abde135ac" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:12:30.048733 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.048705    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "7068fb4a-e8f2-4262-9873-da9abde135ac" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:12:30.048814 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.048787    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7068fb4a-e8f2-4262-9873-da9abde135ac" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:12:30.050374 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.050349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7068fb4a-e8f2-4262-9873-da9abde135ac" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:12:30.056195 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.056170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7068fb4a-e8f2-4262-9873-da9abde135ac-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7068fb4a-e8f2-4262-9873-da9abde135ac" (UID: "7068fb4a-e8f2-4262-9873-da9abde135ac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:12:30.146310 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.146272    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7068fb4a-e8f2-4262-9873-da9abde135ac-ca-trust-extracted\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:12:30.146310 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.146308    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-registry-tls\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:12:30.146547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.146322    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-bound-sa-token\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:12:30.146547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.146335    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-installation-pull-secrets\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:12:30.146547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.146348    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7068fb4a-e8f2-4262-9873-da9abde135ac-image-registry-private-configuration\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:12:30.146547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.146361    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t56kz\" (UniqueName: \"kubernetes.io/projected/7068fb4a-e8f2-4262-9873-da9abde135ac-kube-api-access-t56kz\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:12:30.146547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.146373    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7068fb4a-e8f2-4262-9873-da9abde135ac-trusted-ca\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:12:30.349956 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.349920    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5ffb6d446d-nb8db"]
Jul 28 17:12:30.353381 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:30.353355    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5ffb6d446d-nb8db"]
Jul 28 17:12:31.036815 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:31.036719    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n" event={"ID":"e62fc7a6-3156-4e2d-a376-1a753ef8eb8d","Type":"ContainerStarted","Data":"028507ce867354f484af563b9d57ce7a88284ab7072486266731642f340c2b05"}
Jul 28 17:12:32.319489 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:32.319447    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7068fb4a-e8f2-4262-9873-da9abde135ac" path="/var/lib/kubelet/pods/7068fb4a-e8f2-4262-9873-da9abde135ac/volumes"
Jul 28 17:12:35.054574 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:35.054543    2579 generic.go:356] "Generic (PLEG): container finished" podID="b3e259f7-b62e-4579-8db5-c423186f2510" containerID="42ab1bc32113851955f6677f31397e06129407206be48365337edcc39a47c18d" exitCode=0
Jul 28 17:12:35.055068 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:35.054619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx" event={"ID":"b3e259f7-b62e-4579-8db5-c423186f2510","Type":"ContainerDied","Data":"42ab1bc32113851955f6677f31397e06129407206be48365337edcc39a47c18d"}
Jul 28 17:12:35.055068 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:35.054964    2579 scope.go:117] "RemoveContainer" containerID="42ab1bc32113851955f6677f31397e06129407206be48365337edcc39a47c18d"
Jul 28 17:12:35.056226 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:35.056201    2579 generic.go:356] "Generic (PLEG): container finished" podID="38193ea1-7cb0-4c1e-9664-a9b02b706559" containerID="00e66e035d1cb751c8b0b181db76239d094c28250a42803ac8763f5d0fd7701e" exitCode=0
Jul 28 17:12:35.056332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:35.056236    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zxbld" event={"ID":"38193ea1-7cb0-4c1e-9664-a9b02b706559","Type":"ContainerDied","Data":"00e66e035d1cb751c8b0b181db76239d094c28250a42803ac8763f5d0fd7701e"}
Jul 28 17:12:35.056502 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:35.056489    2579 scope.go:117] "RemoveContainer" containerID="00e66e035d1cb751c8b0b181db76239d094c28250a42803ac8763f5d0fd7701e"
Jul 28 17:12:36.062241 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:36.062205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-6pcqx" event={"ID":"b3e259f7-b62e-4579-8db5-c423186f2510","Type":"ContainerStarted","Data":"aed26c3fa5f0df78387d9bf64dfcd19001b5b801e5a09c007f067bc7f72eb15f"}
Jul 28 17:12:36.063916 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:12:36.063884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-zxbld" event={"ID":"38193ea1-7cb0-4c1e-9664-a9b02b706559","Type":"ContainerStarted","Data":"c8ce99b91167f141c118769f0a41b6351d5723d4b2aab39df441b7aa13f8369b"}
Jul 28 17:13:02.687504 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:13:02.687457    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:02.703127 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:13:02.703103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:13:03.194354 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:13:03.194328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:15:12.160966 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:15:12.160933    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:15:12.167162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:15:12.167137    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:15:12.173497 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:15:12.173476    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:17:59.127114 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.127017    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lf2mh"]
Jul 28 17:17:59.127558 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.127394    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7068fb4a-e8f2-4262-9873-da9abde135ac" containerName="registry"
Jul 28 17:17:59.127558 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.127407    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7068fb4a-e8f2-4262-9873-da9abde135ac" containerName="registry"
Jul 28 17:17:59.127558 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.127462    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7068fb4a-e8f2-4262-9873-da9abde135ac" containerName="registry"
Jul 28 17:17:59.129716 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.129700    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh"
Jul 28 17:17:59.132274 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.132253    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-q5q76\""
Jul 28 17:17:59.132351 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.132294    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 17:17:59.133301 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.133286    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 17:17:59.139858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.139840    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lf2mh"]
Jul 28 17:17:59.240538 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.240502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7vq\" (UniqueName: \"kubernetes.io/projected/c0e2c627-f825-478f-a4f6-a2a6d392a809-kube-api-access-qn7vq\") pod \"cert-manager-webhook-6486ffd997-lf2mh\" (UID: \"c0e2c627-f825-478f-a4f6-a2a6d392a809\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh"
Jul 28 17:17:59.240707 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.240576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0e2c627-f825-478f-a4f6-a2a6d392a809-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lf2mh\" (UID: \"c0e2c627-f825-478f-a4f6-a2a6d392a809\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh"
Jul 28 17:17:59.340989 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.340959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0e2c627-f825-478f-a4f6-a2a6d392a809-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lf2mh\" (UID: \"c0e2c627-f825-478f-a4f6-a2a6d392a809\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh"
Jul 28 17:17:59.341173 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.341010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7vq\" (UniqueName: \"kubernetes.io/projected/c0e2c627-f825-478f-a4f6-a2a6d392a809-kube-api-access-qn7vq\") pod \"cert-manager-webhook-6486ffd997-lf2mh\" (UID: \"c0e2c627-f825-478f-a4f6-a2a6d392a809\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh"
Jul 28 17:17:59.350556 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.350527    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0e2c627-f825-478f-a4f6-a2a6d392a809-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-lf2mh\" (UID: \"c0e2c627-f825-478f-a4f6-a2a6d392a809\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh"
Jul 28 17:17:59.350652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.350617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7vq\" (UniqueName: \"kubernetes.io/projected/c0e2c627-f825-478f-a4f6-a2a6d392a809-kube-api-access-qn7vq\") pod \"cert-manager-webhook-6486ffd997-lf2mh\" (UID: \"c0e2c627-f825-478f-a4f6-a2a6d392a809\") " pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh"
Jul 28 17:17:59.447441 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.447381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh"
Jul 28 17:17:59.572879 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.572858    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-lf2mh"]
Jul 28 17:17:59.575047 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:17:59.575019    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0e2c627_f825_478f_a4f6_a2a6d392a809.slice/crio-f1b655cbaab167ec8f4c419c55757100fbebd6d2f51ca1dcaffeb6f7c30dcc56 WatchSource:0}: Error finding container f1b655cbaab167ec8f4c419c55757100fbebd6d2f51ca1dcaffeb6f7c30dcc56: Status 404 returned error can't find the container with id f1b655cbaab167ec8f4c419c55757100fbebd6d2f51ca1dcaffeb6f7c30dcc56
Jul 28 17:17:59.576929 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:17:59.576912    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:18:00.368319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:00.368284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh" event={"ID":"c0e2c627-f825-478f-a4f6-a2a6d392a809","Type":"ContainerStarted","Data":"f1b655cbaab167ec8f4c419c55757100fbebd6d2f51ca1dcaffeb6f7c30dcc56"}
Jul 28 17:18:03.384279 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:03.384246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh" event={"ID":"c0e2c627-f825-478f-a4f6-a2a6d392a809","Type":"ContainerStarted","Data":"3b9747e0baa61a27458c6ca488f0cab6df8b9125c101636eb9562c62bac0d017"}
Jul 28 17:18:03.384736 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:03.384361    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh"
Jul 28 17:18:03.403229 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:03.403178    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh" podStartSLOduration=1.05879429 podStartE2EDuration="4.403164059s" totalImagesPullingTime="3.344369769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:03.402456878 +0000 UTC m=+471.631655973" watchObservedRunningTime="2026-07-28 17:18:03.403164059 +0000 UTC m=+471.632363154"
Jul 28 17:18:09.391426 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:09.391388    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-lf2mh"
Jul 28 17:18:16.165787 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.165752    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"]
Jul 28 17:18:16.168277 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.168262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"
Jul 28 17:18:16.170981 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.170958    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-pnqgz\""
Jul 28 17:18:16.171067 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.170999    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:18:16.172117 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.172092    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 17:18:16.182197 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.182176    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"]
Jul 28 17:18:16.283485 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.283457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306ff766-a0c0-48bb-bc14-c972c472d504-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4tnc8\" (UID: \"306ff766-a0c0-48bb-bc14-c972c472d504\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"
Jul 28 17:18:16.283598 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.283500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s5bv\" (UniqueName: \"kubernetes.io/projected/306ff766-a0c0-48bb-bc14-c972c472d504-kube-api-access-6s5bv\") pod \"openshift-lws-operator-fd8ccff4c-4tnc8\" (UID: \"306ff766-a0c0-48bb-bc14-c972c472d504\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"
Jul 28 17:18:16.383931 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.383903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s5bv\" (UniqueName: \"kubernetes.io/projected/306ff766-a0c0-48bb-bc14-c972c472d504-kube-api-access-6s5bv\") pod \"openshift-lws-operator-fd8ccff4c-4tnc8\" (UID: \"306ff766-a0c0-48bb-bc14-c972c472d504\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"
Jul 28 17:18:16.384068 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.383969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306ff766-a0c0-48bb-bc14-c972c472d504-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4tnc8\" (UID: \"306ff766-a0c0-48bb-bc14-c972c472d504\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"
Jul 28 17:18:16.384279 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.384261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/306ff766-a0c0-48bb-bc14-c972c472d504-tmp\") pod \"openshift-lws-operator-fd8ccff4c-4tnc8\" (UID: \"306ff766-a0c0-48bb-bc14-c972c472d504\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"
Jul 28 17:18:16.394390 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.394367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s5bv\" (UniqueName: \"kubernetes.io/projected/306ff766-a0c0-48bb-bc14-c972c472d504-kube-api-access-6s5bv\") pod \"openshift-lws-operator-fd8ccff4c-4tnc8\" (UID: \"306ff766-a0c0-48bb-bc14-c972c472d504\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"
Jul 28 17:18:16.477546 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.477492    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"
Jul 28 17:18:16.603394 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:16.603370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8"]
Jul 28 17:18:16.605631 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:18:16.605600    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod306ff766_a0c0_48bb_bc14_c972c472d504.slice/crio-ab20812ea81ae7d0691fcbbe119dcf235993574368d2e2bdcc01466af30294ed WatchSource:0}: Error finding container ab20812ea81ae7d0691fcbbe119dcf235993574368d2e2bdcc01466af30294ed: Status 404 returned error can't find the container with id ab20812ea81ae7d0691fcbbe119dcf235993574368d2e2bdcc01466af30294ed
Jul 28 17:18:17.440900 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:17.440869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8" event={"ID":"306ff766-a0c0-48bb-bc14-c972c472d504","Type":"ContainerStarted","Data":"ab20812ea81ae7d0691fcbbe119dcf235993574368d2e2bdcc01466af30294ed"}
Jul 28 17:18:21.460550 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:21.460510    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8" event={"ID":"306ff766-a0c0-48bb-bc14-c972c472d504","Type":"ContainerStarted","Data":"ace480864963a063e6da31b93d2520ae381c49b032128529ee2d59661b0e31ea"}
Jul 28 17:18:21.479621 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:21.479573    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-4tnc8" podStartSLOduration=1.089264556 podStartE2EDuration="5.479554302s" totalImagesPullingTime="4.390289746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:21.47946585 +0000 UTC m=+489.708664961" watchObservedRunningTime="2026-07-28 17:18:21.479554302 +0000 UTC m=+489.708753400"
Jul 28 17:18:30.608984 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.608955    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"]
Jul 28 17:18:30.613776 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.613757    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.617972 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.617937    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 17:18:30.618115 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.617969    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-mfs9z\""
Jul 28 17:18:30.618115 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.618007    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 17:18:30.618115 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.618012    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 17:18:30.624608 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.624587    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"]
Jul 28 17:18:30.699221 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.699198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5755e1cd-4dd5-428b-800d-92a8661b1b92-manager-config\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.699348 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.699237    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5755e1cd-4dd5-428b-800d-92a8661b1b92-cert\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.699348 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.699271    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5755e1cd-4dd5-428b-800d-92a8661b1b92-metrics-cert\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.699348 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.699315    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tfh8\" (UniqueName: \"kubernetes.io/projected/5755e1cd-4dd5-428b-800d-92a8661b1b92-kube-api-access-4tfh8\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.800739 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.800703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5755e1cd-4dd5-428b-800d-92a8661b1b92-manager-config\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.800739 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.800745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5755e1cd-4dd5-428b-800d-92a8661b1b92-cert\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.800946 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.800792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5755e1cd-4dd5-428b-800d-92a8661b1b92-metrics-cert\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.800946 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.800824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tfh8\" (UniqueName: \"kubernetes.io/projected/5755e1cd-4dd5-428b-800d-92a8661b1b92-kube-api-access-4tfh8\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.801495 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.801471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5755e1cd-4dd5-428b-800d-92a8661b1b92-manager-config\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.803370 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.803343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5755e1cd-4dd5-428b-800d-92a8661b1b92-cert\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.803493 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.803445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5755e1cd-4dd5-428b-800d-92a8661b1b92-metrics-cert\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.814365 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.814345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tfh8\" (UniqueName: \"kubernetes.io/projected/5755e1cd-4dd5-428b-800d-92a8661b1b92-kube-api-access-4tfh8\") pod \"lws-controller-manager-659b67c655-hmff7\" (UID: \"5755e1cd-4dd5-428b-800d-92a8661b1b92\") " pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:30.923278 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:30.923194    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:31.054065 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:31.054041    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"]
Jul 28 17:18:31.056630 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:18:31.056604    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5755e1cd_4dd5_428b_800d_92a8661b1b92.slice/crio-40a930a3269ae3b41042facd59eccedfc63ed7c9c67dd62604ce05a3063b5ec5 WatchSource:0}: Error finding container 40a930a3269ae3b41042facd59eccedfc63ed7c9c67dd62604ce05a3063b5ec5: Status 404 returned error can't find the container with id 40a930a3269ae3b41042facd59eccedfc63ed7c9c67dd62604ce05a3063b5ec5
Jul 28 17:18:31.505599 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:31.505569    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7" event={"ID":"5755e1cd-4dd5-428b-800d-92a8661b1b92","Type":"ContainerStarted","Data":"40a930a3269ae3b41042facd59eccedfc63ed7c9c67dd62604ce05a3063b5ec5"}
Jul 28 17:18:36.530502 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:36.530467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7" event={"ID":"5755e1cd-4dd5-428b-800d-92a8661b1b92","Type":"ContainerStarted","Data":"219f7b579da519b4bb14660eb0b458584f2357dbd3db40ce2147a85cb9ff8071"}
Jul 28 17:18:36.530892 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:36.530545    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:18:36.553426 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:36.553349    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7" podStartSLOduration=1.307881713 podStartE2EDuration="6.553334429s" totalImagesPullingTime="5.245452716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:36.551425249 +0000 UTC m=+504.780624346" watchObservedRunningTime="2026-07-28 17:18:36.553334429 +0000 UTC m=+504.782533551"
Jul 28 17:18:47.537763 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:18:47.537733    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-659b67c655-hmff7"
Jul 28 17:19:10.812897 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:10.812816    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9"]
Jul 28 17:19:10.815662 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:10.815642    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9"
Jul 28 17:19:10.819240 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:10.819215    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 17:19:10.819630 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:10.819615    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 17:19:10.819712 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:10.819645    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nmhrq\""
Jul 28 17:19:10.829317 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:10.829298    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9"]
Jul 28 17:19:10.941986 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:10.941936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkwx6\" (UniqueName: \"kubernetes.io/projected/58bffa62-0706-48f1-b219-ab028f6934d5-kube-api-access-mkwx6\") pod \"limitador-operator-controller-manager-774d94476f-l7cg9\" (UID: \"58bffa62-0706-48f1-b219-ab028f6934d5\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9"
Jul 28 17:19:11.042515 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:11.042487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkwx6\" (UniqueName: \"kubernetes.io/projected/58bffa62-0706-48f1-b219-ab028f6934d5-kube-api-access-mkwx6\") pod \"limitador-operator-controller-manager-774d94476f-l7cg9\" (UID: \"58bffa62-0706-48f1-b219-ab028f6934d5\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9"
Jul 28 17:19:11.051443 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:11.051421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkwx6\" (UniqueName: \"kubernetes.io/projected/58bffa62-0706-48f1-b219-ab028f6934d5-kube-api-access-mkwx6\") pod \"limitador-operator-controller-manager-774d94476f-l7cg9\" (UID: \"58bffa62-0706-48f1-b219-ab028f6934d5\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9"
Jul 28 17:19:11.125486 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:11.125453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9"
Jul 28 17:19:11.250780 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:11.250750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9"]
Jul 28 17:19:11.253424 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:19:11.253398    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bffa62_0706_48f1_b219_ab028f6934d5.slice/crio-c7112803bc94f414aff95ab13e4fb86a0ca042bab3d46238077ff2793b93bbfe WatchSource:0}: Error finding container c7112803bc94f414aff95ab13e4fb86a0ca042bab3d46238077ff2793b93bbfe: Status 404 returned error can't find the container with id c7112803bc94f414aff95ab13e4fb86a0ca042bab3d46238077ff2793b93bbfe
Jul 28 17:19:11.689812 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:11.689779    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9" event={"ID":"58bffa62-0706-48f1-b219-ab028f6934d5","Type":"ContainerStarted","Data":"c7112803bc94f414aff95ab13e4fb86a0ca042bab3d46238077ff2793b93bbfe"}
Jul 28 17:19:14.706346 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:14.706306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9" event={"ID":"58bffa62-0706-48f1-b219-ab028f6934d5","Type":"ContainerStarted","Data":"08692afac082eddd9f8d16d5e665273b529afa4917998e78a33a391ceeb5fe5d"}
Jul 28 17:19:14.706723 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:14.706391    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9"
Jul 28 17:19:14.735267 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:14.735213    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9" podStartSLOduration=2.167572601 podStartE2EDuration="4.735198761s" totalImagesPullingTime="2.56762616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:14.732500605 +0000 UTC m=+542.961699720" watchObservedRunningTime="2026-07-28 17:19:14.735198761 +0000 UTC m=+542.964397857"
Jul 28 17:19:25.712776 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:25.712745    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-l7cg9"
Jul 28 17:19:57.658468 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.658436    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"]
Jul 28 17:19:57.663682 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.663660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:19:57.666553 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.666534    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 17:19:57.666692 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.666674    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hg4cd\""
Jul 28 17:19:57.673349 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.673327    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"]
Jul 28 17:19:57.747310 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.747278    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"]
Jul 28 17:19:57.764652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.764618    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dngmc\" (UniqueName: \"kubernetes.io/projected/8c42056d-39e9-4514-a80e-5ffe8f540024-kube-api-access-dngmc\") pod \"limitador-limitador-859c4d96d6-5hfqc\" (UID: \"8c42056d-39e9-4514-a80e-5ffe8f540024\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:19:57.764833 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.764658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c42056d-39e9-4514-a80e-5ffe8f540024-config-file\") pod \"limitador-limitador-859c4d96d6-5hfqc\" (UID: \"8c42056d-39e9-4514-a80e-5ffe8f540024\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:19:57.865416 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.865378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dngmc\" (UniqueName: \"kubernetes.io/projected/8c42056d-39e9-4514-a80e-5ffe8f540024-kube-api-access-dngmc\") pod \"limitador-limitador-859c4d96d6-5hfqc\" (UID: \"8c42056d-39e9-4514-a80e-5ffe8f540024\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:19:57.865618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.865430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c42056d-39e9-4514-a80e-5ffe8f540024-config-file\") pod \"limitador-limitador-859c4d96d6-5hfqc\" (UID: \"8c42056d-39e9-4514-a80e-5ffe8f540024\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:19:57.866117 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.866065    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c42056d-39e9-4514-a80e-5ffe8f540024-config-file\") pod \"limitador-limitador-859c4d96d6-5hfqc\" (UID: \"8c42056d-39e9-4514-a80e-5ffe8f540024\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:19:57.874190 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.874168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dngmc\" (UniqueName: \"kubernetes.io/projected/8c42056d-39e9-4514-a80e-5ffe8f540024-kube-api-access-dngmc\") pod \"limitador-limitador-859c4d96d6-5hfqc\" (UID: \"8c42056d-39e9-4514-a80e-5ffe8f540024\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:19:57.974398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:57.974311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:19:58.109891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:58.109849    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"]
Jul 28 17:19:58.113653 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:19:58.113624    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c42056d_39e9_4514_a80e_5ffe8f540024.slice/crio-3e9ec849ec8f2cbb45ee9fdb7441bea4158ecacb08fe7107eda39ef80e0dd3da WatchSource:0}: Error finding container 3e9ec849ec8f2cbb45ee9fdb7441bea4158ecacb08fe7107eda39ef80e0dd3da: Status 404 returned error can't find the container with id 3e9ec849ec8f2cbb45ee9fdb7441bea4158ecacb08fe7107eda39ef80e0dd3da
Jul 28 17:19:58.907251 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:58.907211    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc" event={"ID":"8c42056d-39e9-4514-a80e-5ffe8f540024","Type":"ContainerStarted","Data":"3e9ec849ec8f2cbb45ee9fdb7441bea4158ecacb08fe7107eda39ef80e0dd3da"}
Jul 28 17:19:59.913603 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:59.913570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc" event={"ID":"8c42056d-39e9-4514-a80e-5ffe8f540024","Type":"ContainerStarted","Data":"7c04719acc0a6de1bff70ef3955315b977344bc09ea1eedb8d44360374d15f2d"}
Jul 28 17:19:59.914002 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:59.913647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:19:59.933518 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:19:59.933465    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc" podStartSLOduration=1.610905788 podStartE2EDuration="2.933450689s" totalImagesPullingTime="1.322544901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:59.931960648 +0000 UTC m=+588.161159750" watchObservedRunningTime="2026-07-28 17:19:59.933450689 +0000 UTC m=+588.162649851"
Jul 28 17:20:10.919359 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:10.919328    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:20:11.424284 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.424248    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"]
Jul 28 17:20:11.424452 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.424414    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc" podUID="8c42056d-39e9-4514-a80e-5ffe8f540024" containerName="limitador" containerID="cri-o://7c04719acc0a6de1bff70ef3955315b977344bc09ea1eedb8d44360374d15f2d" gracePeriod=30
Jul 28 17:20:11.969260 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.969234    2579 generic.go:356] "Generic (PLEG): container finished" podID="8c42056d-39e9-4514-a80e-5ffe8f540024" containerID="7c04719acc0a6de1bff70ef3955315b977344bc09ea1eedb8d44360374d15f2d" exitCode=0
Jul 28 17:20:11.969578 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.969309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc" event={"ID":"8c42056d-39e9-4514-a80e-5ffe8f540024","Type":"ContainerDied","Data":"7c04719acc0a6de1bff70ef3955315b977344bc09ea1eedb8d44360374d15f2d"}
Jul 28 17:20:11.969578 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.969376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc" event={"ID":"8c42056d-39e9-4514-a80e-5ffe8f540024","Type":"ContainerDied","Data":"3e9ec849ec8f2cbb45ee9fdb7441bea4158ecacb08fe7107eda39ef80e0dd3da"}
Jul 28 17:20:11.969578 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.969404    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9ec849ec8f2cbb45ee9fdb7441bea4158ecacb08fe7107eda39ef80e0dd3da"
Jul 28 17:20:11.973346 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.973328    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:20:11.998798 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.998723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c42056d-39e9-4514-a80e-5ffe8f540024-config-file\") pod \"8c42056d-39e9-4514-a80e-5ffe8f540024\" (UID: \"8c42056d-39e9-4514-a80e-5ffe8f540024\") "
Jul 28 17:20:11.998921 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.998875    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dngmc\" (UniqueName: \"kubernetes.io/projected/8c42056d-39e9-4514-a80e-5ffe8f540024-kube-api-access-dngmc\") pod \"8c42056d-39e9-4514-a80e-5ffe8f540024\" (UID: \"8c42056d-39e9-4514-a80e-5ffe8f540024\") "
Jul 28 17:20:11.999060 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.999039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c42056d-39e9-4514-a80e-5ffe8f540024-config-file" (OuterVolumeSpecName: "config-file") pod "8c42056d-39e9-4514-a80e-5ffe8f540024" (UID: "8c42056d-39e9-4514-a80e-5ffe8f540024"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:20:11.999158 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:11.999145    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8c42056d-39e9-4514-a80e-5ffe8f540024-config-file\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:20:12.001255 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:12.001228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c42056d-39e9-4514-a80e-5ffe8f540024-kube-api-access-dngmc" (OuterVolumeSpecName: "kube-api-access-dngmc") pod "8c42056d-39e9-4514-a80e-5ffe8f540024" (UID: "8c42056d-39e9-4514-a80e-5ffe8f540024"). InnerVolumeSpecName "kube-api-access-dngmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:20:12.100445 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:12.100410    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dngmc\" (UniqueName: \"kubernetes.io/projected/8c42056d-39e9-4514-a80e-5ffe8f540024-kube-api-access-dngmc\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:20:12.196664 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:12.196634    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:20:12.198897 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:12.198876    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:20:12.974519 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:12.974487    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"
Jul 28 17:20:12.993793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:12.993765    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"]
Jul 28 17:20:12.997583 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:12.997559    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-5hfqc"]
Jul 28 17:20:14.320640 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:14.320610    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c42056d-39e9-4514-a80e-5ffe8f540024" path="/var/lib/kubelet/pods/8c42056d-39e9-4514-a80e-5ffe8f540024/volumes"
Jul 28 17:20:28.289444 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.289410    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["observability/jaeger-6bbd7d4c9d-ftlgf"]
Jul 28 17:20:28.289826 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.289809    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c42056d-39e9-4514-a80e-5ffe8f540024" containerName="limitador"
Jul 28 17:20:28.289826 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.289827    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c42056d-39e9-4514-a80e-5ffe8f540024" containerName="limitador"
Jul 28 17:20:28.289911 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.289901    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c42056d-39e9-4514-a80e-5ffe8f540024" containerName="limitador"
Jul 28 17:20:28.293818 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.293802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-ftlgf"
Jul 28 17:20:28.297603 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.297578    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"openshift-service-ca.crt\""
Jul 28 17:20:28.297710 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.297659    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"observability\"/\"kube-root-ca.crt\""
Jul 28 17:20:28.298615 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.298598    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"observability\"/\"jaeger-dockercfg-r7r28\""
Jul 28 17:20:28.309935 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.309911    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-ftlgf"]
Jul 28 17:20:28.341050 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.341020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx9wd\" (UniqueName: \"kubernetes.io/projected/d587dc3b-a0ec-4bb0-a207-68f7e4cc311d-kube-api-access-zx9wd\") pod \"jaeger-6bbd7d4c9d-ftlgf\" (UID: \"d587dc3b-a0ec-4bb0-a207-68f7e4cc311d\") " pod="observability/jaeger-6bbd7d4c9d-ftlgf"
Jul 28 17:20:28.442224 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.442188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx9wd\" (UniqueName: \"kubernetes.io/projected/d587dc3b-a0ec-4bb0-a207-68f7e4cc311d-kube-api-access-zx9wd\") pod \"jaeger-6bbd7d4c9d-ftlgf\" (UID: \"d587dc3b-a0ec-4bb0-a207-68f7e4cc311d\") " pod="observability/jaeger-6bbd7d4c9d-ftlgf"
Jul 28 17:20:28.451520 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.451494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx9wd\" (UniqueName: \"kubernetes.io/projected/d587dc3b-a0ec-4bb0-a207-68f7e4cc311d-kube-api-access-zx9wd\") pod \"jaeger-6bbd7d4c9d-ftlgf\" (UID: \"d587dc3b-a0ec-4bb0-a207-68f7e4cc311d\") " pod="observability/jaeger-6bbd7d4c9d-ftlgf"
Jul 28 17:20:28.602803 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.602777    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="observability/jaeger-6bbd7d4c9d-ftlgf"
Jul 28 17:20:28.737112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:28.731121    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["observability/jaeger-6bbd7d4c9d-ftlgf"]
Jul 28 17:20:28.737112 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:20:28.733485    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd587dc3b_a0ec_4bb0_a207_68f7e4cc311d.slice/crio-cd385fa57cafc53560f59a6819a1db947330e2e9cc232fbff3774b6e81149d5c WatchSource:0}: Error finding container cd385fa57cafc53560f59a6819a1db947330e2e9cc232fbff3774b6e81149d5c: Status 404 returned error can't find the container with id cd385fa57cafc53560f59a6819a1db947330e2e9cc232fbff3774b6e81149d5c
Jul 28 17:20:29.049187 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:29.049096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-ftlgf" event={"ID":"d587dc3b-a0ec-4bb0-a207-68f7e4cc311d","Type":"ContainerStarted","Data":"cd385fa57cafc53560f59a6819a1db947330e2e9cc232fbff3774b6e81149d5c"}
Jul 28 17:20:32.064413 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:32.064375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="observability/jaeger-6bbd7d4c9d-ftlgf" event={"ID":"d587dc3b-a0ec-4bb0-a207-68f7e4cc311d","Type":"ContainerStarted","Data":"2ec5a7551abf3bd736c6cc673780715b2fa3b972204773cbd96acde58353ecae"}
Jul 28 17:20:32.064822 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:32.064527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="observability/jaeger-6bbd7d4c9d-ftlgf"
Jul 28 17:20:32.066422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:32.066401    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="observability/jaeger-6bbd7d4c9d-ftlgf"
Jul 28 17:20:32.085280 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:20:32.085244    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="observability/jaeger-6bbd7d4c9d-ftlgf" podStartSLOduration=1.672371952 podStartE2EDuration="4.085231596s" totalImagesPullingTime="2.412859644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:32.084226242 +0000 UTC m=+620.313425360" watchObservedRunningTime="2026-07-28 17:20:32.085231596 +0000 UTC m=+620.314430692"
Jul 28 17:21:06.908458 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.908380    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-w4pv6"]
Jul 28 17:21:06.911612 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.911595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:21:06.915867 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.915843    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:21:06.915989 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.915868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-24mbq\""
Jul 28 17:21:06.916906 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.916889    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:21:06.917028 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.916887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 17:21:06.930624 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.930597    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zhv7r"]
Jul 28 17:21:06.934304 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.934281    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-w4pv6"]
Jul 28 17:21:06.934402 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.934381    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:06.937003 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.936989    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-kz4w2\""
Jul 28 17:21:06.937259 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.937245    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 17:21:06.948253 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.948230    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zhv7r"]
Jul 28 17:21:06.980030 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.980002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d7e1a1-93ec-4800-acbe-117455e3e8e8-tls-certs\") pod \"model-serving-api-7b954ffc95-w4pv6\" (UID: \"a4d7e1a1-93ec-4800-acbe-117455e3e8e8\") " pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:21:06.980209 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.980033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hk4m\" (UniqueName: \"kubernetes.io/projected/a4d7e1a1-93ec-4800-acbe-117455e3e8e8-kube-api-access-4hk4m\") pod \"model-serving-api-7b954ffc95-w4pv6\" (UID: \"a4d7e1a1-93ec-4800-acbe-117455e3e8e8\") " pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:21:06.980209 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.980058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a7a30a2-26c8-4d15-89d8-c3fbc38858a0-cert\") pod \"odh-model-controller-75d9cdbddf-zhv7r\" (UID: \"1a7a30a2-26c8-4d15-89d8-c3fbc38858a0\") " pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:06.980209 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:06.980102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km9pr\" (UniqueName: \"kubernetes.io/projected/1a7a30a2-26c8-4d15-89d8-c3fbc38858a0-kube-api-access-km9pr\") pod \"odh-model-controller-75d9cdbddf-zhv7r\" (UID: \"1a7a30a2-26c8-4d15-89d8-c3fbc38858a0\") " pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:07.081537 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.081498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d7e1a1-93ec-4800-acbe-117455e3e8e8-tls-certs\") pod \"model-serving-api-7b954ffc95-w4pv6\" (UID: \"a4d7e1a1-93ec-4800-acbe-117455e3e8e8\") " pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:21:07.081537 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.081548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hk4m\" (UniqueName: \"kubernetes.io/projected/a4d7e1a1-93ec-4800-acbe-117455e3e8e8-kube-api-access-4hk4m\") pod \"model-serving-api-7b954ffc95-w4pv6\" (UID: \"a4d7e1a1-93ec-4800-acbe-117455e3e8e8\") " pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:21:07.081772 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:21:07.081654    2579 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 17:21:07.081772 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.081664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a7a30a2-26c8-4d15-89d8-c3fbc38858a0-cert\") pod \"odh-model-controller-75d9cdbddf-zhv7r\" (UID: \"1a7a30a2-26c8-4d15-89d8-c3fbc38858a0\") " pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:07.081772 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:21:07.081742    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4d7e1a1-93ec-4800-acbe-117455e3e8e8-tls-certs podName:a4d7e1a1-93ec-4800-acbe-117455e3e8e8 nodeName:}" failed. No retries permitted until 2026-07-28 17:21:07.581720663 +0000 UTC m=+655.810919748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a4d7e1a1-93ec-4800-acbe-117455e3e8e8-tls-certs") pod "model-serving-api-7b954ffc95-w4pv6" (UID: "a4d7e1a1-93ec-4800-acbe-117455e3e8e8") : secret "model-serving-api-tls" not found
Jul 28 17:21:07.081772 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.081756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km9pr\" (UniqueName: \"kubernetes.io/projected/1a7a30a2-26c8-4d15-89d8-c3fbc38858a0-kube-api-access-km9pr\") pod \"odh-model-controller-75d9cdbddf-zhv7r\" (UID: \"1a7a30a2-26c8-4d15-89d8-c3fbc38858a0\") " pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:07.081932 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:21:07.081792    2579 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 28 17:21:07.081932 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:21:07.081860    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a7a30a2-26c8-4d15-89d8-c3fbc38858a0-cert podName:1a7a30a2-26c8-4d15-89d8-c3fbc38858a0 nodeName:}" failed. No retries permitted until 2026-07-28 17:21:07.581847665 +0000 UTC m=+655.811046739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1a7a30a2-26c8-4d15-89d8-c3fbc38858a0-cert") pod "odh-model-controller-75d9cdbddf-zhv7r" (UID: "1a7a30a2-26c8-4d15-89d8-c3fbc38858a0") : secret "odh-model-controller-webhook-cert" not found
Jul 28 17:21:07.092406 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.092381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hk4m\" (UniqueName: \"kubernetes.io/projected/a4d7e1a1-93ec-4800-acbe-117455e3e8e8-kube-api-access-4hk4m\") pod \"model-serving-api-7b954ffc95-w4pv6\" (UID: \"a4d7e1a1-93ec-4800-acbe-117455e3e8e8\") " pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:21:07.092850 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.092829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km9pr\" (UniqueName: \"kubernetes.io/projected/1a7a30a2-26c8-4d15-89d8-c3fbc38858a0-kube-api-access-km9pr\") pod \"odh-model-controller-75d9cdbddf-zhv7r\" (UID: \"1a7a30a2-26c8-4d15-89d8-c3fbc38858a0\") " pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:07.586894 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.586860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d7e1a1-93ec-4800-acbe-117455e3e8e8-tls-certs\") pod \"model-serving-api-7b954ffc95-w4pv6\" (UID: \"a4d7e1a1-93ec-4800-acbe-117455e3e8e8\") " pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:21:07.587120 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.586902    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a7a30a2-26c8-4d15-89d8-c3fbc38858a0-cert\") pod \"odh-model-controller-75d9cdbddf-zhv7r\" (UID: \"1a7a30a2-26c8-4d15-89d8-c3fbc38858a0\") " pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:07.589509 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.589480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4d7e1a1-93ec-4800-acbe-117455e3e8e8-tls-certs\") pod \"model-serving-api-7b954ffc95-w4pv6\" (UID: \"a4d7e1a1-93ec-4800-acbe-117455e3e8e8\") " pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:21:07.589616 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.589480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1a7a30a2-26c8-4d15-89d8-c3fbc38858a0-cert\") pod \"odh-model-controller-75d9cdbddf-zhv7r\" (UID: \"1a7a30a2-26c8-4d15-89d8-c3fbc38858a0\") " pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:07.821847 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.821812    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:21:07.843854 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.843798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:07.965399 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.965326    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-w4pv6"]
Jul 28 17:21:07.968415 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:21:07.968385    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d7e1a1_93ec_4800_acbe_117455e3e8e8.slice/crio-665954d2f06d9f0b7d5034e4f43c224b57f895a4c0fab42c37e7bdc5d4c53821 WatchSource:0}: Error finding container 665954d2f06d9f0b7d5034e4f43c224b57f895a4c0fab42c37e7bdc5d4c53821: Status 404 returned error can't find the container with id 665954d2f06d9f0b7d5034e4f43c224b57f895a4c0fab42c37e7bdc5d4c53821
Jul 28 17:21:07.989466 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:07.989440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-zhv7r"]
Jul 28 17:21:07.991302 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:21:07.991279    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7a30a2_26c8_4d15_89d8_c3fbc38858a0.slice/crio-7e708d55d3064a1f84d50e97c130702750ae431a98ef828cd05df489e43c5161 WatchSource:0}: Error finding container 7e708d55d3064a1f84d50e97c130702750ae431a98ef828cd05df489e43c5161: Status 404 returned error can't find the container with id 7e708d55d3064a1f84d50e97c130702750ae431a98ef828cd05df489e43c5161
Jul 28 17:21:08.234895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:08.234799    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-zhv7r" event={"ID":"1a7a30a2-26c8-4d15-89d8-c3fbc38858a0","Type":"ContainerStarted","Data":"7e708d55d3064a1f84d50e97c130702750ae431a98ef828cd05df489e43c5161"}
Jul 28 17:21:08.235854 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:08.235830    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-w4pv6" event={"ID":"a4d7e1a1-93ec-4800-acbe-117455e3e8e8","Type":"ContainerStarted","Data":"665954d2f06d9f0b7d5034e4f43c224b57f895a4c0fab42c37e7bdc5d4c53821"}
Jul 28 17:21:12.261329 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:12.261291    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-zhv7r" event={"ID":"1a7a30a2-26c8-4d15-89d8-c3fbc38858a0","Type":"ContainerStarted","Data":"4c5863b10415e96c5579c1dd058df3b10497532f2ab4b49df575302edf5c2ea6"}
Jul 28 17:21:12.261873 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:12.261402    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:12.262978 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:12.262953    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-w4pv6" event={"ID":"a4d7e1a1-93ec-4800-acbe-117455e3e8e8","Type":"ContainerStarted","Data":"724c0562149bb9a3f24ee1ab84fa4041c5a236a7fac4c7ef0d2de565960b3734"}
Jul 28 17:21:12.263132 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:12.263020    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:21:12.301451 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:12.301409    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-w4pv6" podStartSLOduration=2.64349797 podStartE2EDuration="6.301392236s" totalImagesPullingTime="3.657894266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:12.301168238 +0000 UTC m=+660.530367339" watchObservedRunningTime="2026-07-28 17:21:12.301392236 +0000 UTC m=+660.530591335"
Jul 28 17:21:12.301828 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:12.301797    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-zhv7r" podStartSLOduration=2.618959921 podStartE2EDuration="6.301788662s" totalImagesPullingTime="3.682828741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:12.282831672 +0000 UTC m=+660.512030768" watchObservedRunningTime="2026-07-28 17:21:12.301788662 +0000 UTC m=+660.530987758"
Jul 28 17:21:23.270883 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:23.270850    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-zhv7r"
Jul 28 17:21:23.273034 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:21:23.273014    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-w4pv6"
Jul 28 17:23:16.137548 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.137518    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"]
Jul 28 17:23:16.140620 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.140603    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.144550 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.144524    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Jul 28 17:23:16.144550 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.144546    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Jul 28 17:23:16.144712 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.144558    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-7n7sc\""
Jul 28 17:23:16.144712 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.144524    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Jul 28 17:23:16.151152 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.151132    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"]
Jul 28 17:23:16.309738 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.309708    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.309866 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.309747    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-dshm\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.309866 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.309773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5299f753-3853-4678-b4d7-230596aa0f41-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.309866 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.309835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-home\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.309980 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.309875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr9s7\" (UniqueName: \"kubernetes.io/projected/5299f753-3853-4678-b4d7-230596aa0f41-kube-api-access-nr9s7\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.309980 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.309908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.410915 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.410850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-dshm\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.410915 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.410879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5299f753-3853-4678-b4d7-230596aa0f41-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.410915 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.410899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-home\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.411164 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.410926    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr9s7\" (UniqueName: \"kubernetes.io/projected/5299f753-3853-4678-b4d7-230596aa0f41-kube-api-access-nr9s7\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.411164 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.410957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.411164 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.411029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.411391 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.411362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-home\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.411508 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.411415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.411508 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.411442    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.413185 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.413163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-dshm\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.413309 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.413293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5299f753-3853-4678-b4d7-230596aa0f41-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.420036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.420013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr9s7\" (UniqueName: \"kubernetes.io/projected/5299f753-3853-4678-b4d7-230596aa0f41-kube-api-access-nr9s7\") pod \"fc-smoke-test-kserve-7f4687897-jjkdj\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.451806 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.451785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:16.587166 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.587137    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"]
Jul 28 17:23:16.590486 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:23:16.590458    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5299f753_3853_4678_b4d7_230596aa0f41.slice/crio-ccd34c12e0d8a11e3c0089d3c9d737b812735808a6eaed96d5f818126940b1cb WatchSource:0}: Error finding container ccd34c12e0d8a11e3c0089d3c9d737b812735808a6eaed96d5f818126940b1cb: Status 404 returned error can't find the container with id ccd34c12e0d8a11e3c0089d3c9d737b812735808a6eaed96d5f818126940b1cb
Jul 28 17:23:16.592406 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.592389    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:23:16.873747 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:16.873712    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj" event={"ID":"5299f753-3853-4678-b4d7-230596aa0f41","Type":"ContainerStarted","Data":"ccd34c12e0d8a11e3c0089d3c9d737b812735808a6eaed96d5f818126940b1cb"}
Jul 28 17:23:18.885937 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:18.885902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj" event={"ID":"5299f753-3853-4678-b4d7-230596aa0f41","Type":"ContainerStarted","Data":"9bd5ec8e8f8a0ca33d67e00cd475786b732e93057069c07ce753364c09b1f6fe"}
Jul 28 17:23:18.905739 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:18.905693    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj" podStartSLOduration=1.506271555 podStartE2EDuration="2.905678012s" totalImagesPullingTime="1.399406457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:18.904735395 +0000 UTC m=+787.133934517" watchObservedRunningTime="2026-07-28 17:23:18.905678012 +0000 UTC m=+787.134877108"
Jul 28 17:23:19.451864 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:19.451832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:26.452438 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:26.452409    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:26.464773 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:26.464743    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:23:26.935291 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:23:26.935251    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:24:06.812252 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:06.812169    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"]
Jul 28 17:24:06.821545 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:24:06.821506    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:24:06.821545 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:24:06.821545 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:24:06.821545 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="9bd5ec8e8f8a0ca33d67e00cd475786b732e93057069c07ce753364c09b1f6fe" cmd=["/bin/sleep","15"]
Jul 28 17:24:06.821830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:06.821562    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj" message="" err=<
Jul 28 17:24:06.821830 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:24:06.821830 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:24:06.821830 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:24:06.821830 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:24:06.821574    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:24:06.821830 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:24:06.821830 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:24:06.821830 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj" podUID="5299f753-3853-4678-b4d7-230596aa0f41" containerName="main" containerID="cri-o://9bd5ec8e8f8a0ca33d67e00cd475786b732e93057069c07ce753364c09b1f6fe"
Jul 28 17:24:06.821830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:06.821616    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj" podUID="5299f753-3853-4678-b4d7-230596aa0f41" containerName="main" containerID="cri-o://9bd5ec8e8f8a0ca33d67e00cd475786b732e93057069c07ce753364c09b1f6fe" gracePeriod=60
Jul 28 17:24:06.924557 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:06.924480    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj" podUID="5299f753-3853-4678-b4d7-230596aa0f41" containerName="main" probeResult="failure" output="Get \"https://10.132.0.32:8000/health\": dial tcp 10.132.0.32:8000: connect: connection refused"
Jul 28 17:24:07.117263 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.117228    2579 generic.go:356] "Generic (PLEG): container finished" podID="5299f753-3853-4678-b4d7-230596aa0f41" containerID="9bd5ec8e8f8a0ca33d67e00cd475786b732e93057069c07ce753364c09b1f6fe" exitCode=0
Jul 28 17:24:07.117416 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.117297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj" event={"ID":"5299f753-3853-4678-b4d7-230596aa0f41","Type":"ContainerDied","Data":"9bd5ec8e8f8a0ca33d67e00cd475786b732e93057069c07ce753364c09b1f6fe"}
Jul 28 17:24:07.172887 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.172856    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:24:07.348993 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.348907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nr9s7\" (UniqueName: \"kubernetes.io/projected/5299f753-3853-4678-b4d7-230596aa0f41-kube-api-access-nr9s7\") pod \"5299f753-3853-4678-b4d7-230596aa0f41\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") "
Jul 28 17:24:07.348993 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.348953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-dshm\") pod \"5299f753-3853-4678-b4d7-230596aa0f41\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") "
Jul 28 17:24:07.348993 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.348973    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-home\") pod \"5299f753-3853-4678-b4d7-230596aa0f41\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") "
Jul 28 17:24:07.349318 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.349016    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-tmp-dir\") pod \"5299f753-3853-4678-b4d7-230596aa0f41\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") "
Jul 28 17:24:07.349318 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.349104    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5299f753-3853-4678-b4d7-230596aa0f41-tls-certs\") pod \"5299f753-3853-4678-b4d7-230596aa0f41\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") "
Jul 28 17:24:07.349318 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.349217    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-model-cache\") pod \"5299f753-3853-4678-b4d7-230596aa0f41\" (UID: \"5299f753-3853-4678-b4d7-230596aa0f41\") "
Jul 28 17:24:07.349318 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.349268    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-home" (OuterVolumeSpecName: "home") pod "5299f753-3853-4678-b4d7-230596aa0f41" (UID: "5299f753-3853-4678-b4d7-230596aa0f41"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:24:07.349318 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.349306    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5299f753-3853-4678-b4d7-230596aa0f41" (UID: "5299f753-3853-4678-b4d7-230596aa0f41"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:24:07.349520 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.349460    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:24:07.349520 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.349468    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-model-cache" (OuterVolumeSpecName: "model-cache") pod "5299f753-3853-4678-b4d7-230596aa0f41" (UID: "5299f753-3853-4678-b4d7-230596aa0f41"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:24:07.349520 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.349477    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:24:07.351235 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.351213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-dshm" (OuterVolumeSpecName: "dshm") pod "5299f753-3853-4678-b4d7-230596aa0f41" (UID: "5299f753-3853-4678-b4d7-230596aa0f41"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:24:07.351346 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.351285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5299f753-3853-4678-b4d7-230596aa0f41-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5299f753-3853-4678-b4d7-230596aa0f41" (UID: "5299f753-3853-4678-b4d7-230596aa0f41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:07.351346 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.351294    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5299f753-3853-4678-b4d7-230596aa0f41-kube-api-access-nr9s7" (OuterVolumeSpecName: "kube-api-access-nr9s7") pod "5299f753-3853-4678-b4d7-230596aa0f41" (UID: "5299f753-3853-4678-b4d7-230596aa0f41"). InnerVolumeSpecName "kube-api-access-nr9s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:24:07.450855 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.450824    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr9s7\" (UniqueName: \"kubernetes.io/projected/5299f753-3853-4678-b4d7-230596aa0f41-kube-api-access-nr9s7\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:24:07.450855 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.450854    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:24:07.451037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.450868    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5299f753-3853-4678-b4d7-230596aa0f41-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:24:07.451037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:07.450881    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5299f753-3853-4678-b4d7-230596aa0f41-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:24:08.123522 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:08.123489    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"
Jul 28 17:24:08.123945 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:08.123523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj" event={"ID":"5299f753-3853-4678-b4d7-230596aa0f41","Type":"ContainerDied","Data":"ccd34c12e0d8a11e3c0089d3c9d737b812735808a6eaed96d5f818126940b1cb"}
Jul 28 17:24:08.123945 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:08.123586    2579 scope.go:117] "RemoveContainer" containerID="9bd5ec8e8f8a0ca33d67e00cd475786b732e93057069c07ce753364c09b1f6fe"
Jul 28 17:24:08.145833 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:08.145803    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"]
Jul 28 17:24:08.149852 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:08.149831    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-jjkdj"]
Jul 28 17:24:08.319745 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:08.319715    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5299f753-3853-4678-b4d7-230596aa0f41" path="/var/lib/kubelet/pods/5299f753-3853-4678-b4d7-230596aa0f41/volumes"
Jul 28 17:24:41.926940 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:41.926910    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"]
Jul 28 17:24:41.927554 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:41.927358    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5299f753-3853-4678-b4d7-230596aa0f41" containerName="main"
Jul 28 17:24:41.927554 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:41.927372    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5299f753-3853-4678-b4d7-230596aa0f41" containerName="main"
Jul 28 17:24:41.927554 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:41.927430    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5299f753-3853-4678-b4d7-230596aa0f41" containerName="main"
Jul 28 17:24:41.930366 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:41.930348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:41.933601 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:41.933583    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Jul 28 17:24:41.933719 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:41.933583    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-b4zzn\""
Jul 28 17:24:41.934538 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:41.934516    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Jul 28 17:24:41.934695 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:41.934675    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Jul 28 17:24:41.941582 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:41.941561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"]
Jul 28 17:24:42.066859 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.066829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.067009 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.066909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-home\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.067009 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.066943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.067009 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.066970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.067009 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.066990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r255m\" (UniqueName: \"kubernetes.io/projected/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-kube-api-access-r255m\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.067161 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.067014    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.168001 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.167966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.168209 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.168097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-home\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.168209 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.168148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.168335 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.168286    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.168391 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.168344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r255m\" (UniqueName: \"kubernetes.io/projected/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-kube-api-access-r255m\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.168391 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.168378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.168572 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.168548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.168764 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.168720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.168996 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.168980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-home\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.170910 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.170885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.171199 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.171179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.179451 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.179399    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r255m\" (UniqueName: \"kubernetes.io/projected/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-kube-api-access-r255m\") pod \"fc-concurrency-test-kserve-b5d44467c-9dmfp\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.240797 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.240766    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:42.373390 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:42.373365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"]
Jul 28 17:24:42.375326 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:24:42.375299    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdd77b6_3f47_4a14_83c1_b38e44b6e497.slice/crio-8282c6e31abcf88b45063c76bcd70d69abe1c2efe9ecc83d851612a00f15abdb WatchSource:0}: Error finding container 8282c6e31abcf88b45063c76bcd70d69abe1c2efe9ecc83d851612a00f15abdb: Status 404 returned error can't find the container with id 8282c6e31abcf88b45063c76bcd70d69abe1c2efe9ecc83d851612a00f15abdb
Jul 28 17:24:43.294322 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:43.294287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp" event={"ID":"bfdd77b6-3f47-4a14-83c1-b38e44b6e497","Type":"ContainerStarted","Data":"4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd"}
Jul 28 17:24:43.294751 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:43.294341    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp" event={"ID":"bfdd77b6-3f47-4a14-83c1-b38e44b6e497","Type":"ContainerStarted","Data":"8282c6e31abcf88b45063c76bcd70d69abe1c2efe9ecc83d851612a00f15abdb"}
Jul 28 17:24:43.312006 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:43.311958    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp" podStartSLOduration=2.311943113 podStartE2EDuration="2.311943113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:43.311326045 +0000 UTC m=+871.540525162" watchObservedRunningTime="2026-07-28 17:24:43.311943113 +0000 UTC m=+871.541142210"
Jul 28 17:24:44.240885 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:44.240856    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:52.241151 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:52.241118    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:52.253044 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:52.253017    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:24:52.348834 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:24:52.348807    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:25:12.229136 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:12.229040    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:25:12.229136 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:12.229066    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:25:23.067578 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.067544    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"]
Jul 28 17:25:23.076672 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:25:23.076630    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:25:23.076672 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:25:23.076672 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:25:23.076672 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd" cmd=["/bin/sleep","15"]
Jul 28 17:25:23.076915 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.076690    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp" message="" err=<
Jul 28 17:25:23.076915 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:25:23.076915 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:25:23.076915 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:25:23.076915 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:25:23.076707    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:25:23.076915 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:25:23.076915 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:25:23.076915 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp" podUID="bfdd77b6-3f47-4a14-83c1-b38e44b6e497" containerName="main" containerID="cri-o://4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd"
Jul 28 17:25:23.076915 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.076751    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp" podUID="bfdd77b6-3f47-4a14-83c1-b38e44b6e497" containerName="main" containerID="cri-o://4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd" gracePeriod=60
Jul 28 17:25:23.338739 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.338663    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp" podUID="bfdd77b6-3f47-4a14-83c1-b38e44b6e497" containerName="main" probeResult="failure" output="Get \"https://10.132.0.33:8000/health\": dial tcp 10.132.0.33:8000: connect: connection refused"
Jul 28 17:25:23.430999 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.430976    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:25:23.491266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.491221    2579 generic.go:356] "Generic (PLEG): container finished" podID="bfdd77b6-3f47-4a14-83c1-b38e44b6e497" containerID="4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd" exitCode=0
Jul 28 17:25:23.491388 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.491273    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"
Jul 28 17:25:23.491388 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.491275    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp" event={"ID":"bfdd77b6-3f47-4a14-83c1-b38e44b6e497","Type":"ContainerDied","Data":"4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd"}
Jul 28 17:25:23.491388 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.491340    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp" event={"ID":"bfdd77b6-3f47-4a14-83c1-b38e44b6e497","Type":"ContainerDied","Data":"8282c6e31abcf88b45063c76bcd70d69abe1c2efe9ecc83d851612a00f15abdb"}
Jul 28 17:25:23.491388 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.491368    2579 scope.go:117] "RemoveContainer" containerID="4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd"
Jul 28 17:25:23.502140 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.502068    2579 scope.go:117] "RemoveContainer" containerID="4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd"
Jul 28 17:25:23.502455 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:25:23.502436    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd\": container with ID starting with 4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd not found: ID does not exist" containerID="4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd"
Jul 28 17:25:23.502523 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.502467    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd"} err="failed to get container status \"4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd\": rpc error: code = NotFound desc = could not find container \"4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd\": container with ID starting with 4188c32cc819e2b000d877099bea6622c2c18cd728dd79b6af0707cc70d01cdd not found: ID does not exist"
Jul 28 17:25:23.519303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.519280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-dshm\") pod \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") "
Jul 28 17:25:23.519399 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.519312    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tmp-dir\") pod \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") "
Jul 28 17:25:23.519399 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.519332    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tls-certs\") pod \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") "
Jul 28 17:25:23.519399 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.519384    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r255m\" (UniqueName: \"kubernetes.io/projected/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-kube-api-access-r255m\") pod \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") "
Jul 28 17:25:23.519530 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.519478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-home\") pod \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") "
Jul 28 17:25:23.519530 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.519509    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-model-cache\") pod \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\" (UID: \"bfdd77b6-3f47-4a14-83c1-b38e44b6e497\") "
Jul 28 17:25:23.519632 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.519553    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bfdd77b6-3f47-4a14-83c1-b38e44b6e497" (UID: "bfdd77b6-3f47-4a14-83c1-b38e44b6e497"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:23.519802 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.519775    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-home" (OuterVolumeSpecName: "home") pod "bfdd77b6-3f47-4a14-83c1-b38e44b6e497" (UID: "bfdd77b6-3f47-4a14-83c1-b38e44b6e497"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:23.519890 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.519812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-model-cache" (OuterVolumeSpecName: "model-cache") pod "bfdd77b6-3f47-4a14-83c1-b38e44b6e497" (UID: "bfdd77b6-3f47-4a14-83c1-b38e44b6e497"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:23.519890 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.519831    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:23.521506 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.521486    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-kube-api-access-r255m" (OuterVolumeSpecName: "kube-api-access-r255m") pod "bfdd77b6-3f47-4a14-83c1-b38e44b6e497" (UID: "bfdd77b6-3f47-4a14-83c1-b38e44b6e497"). InnerVolumeSpecName "kube-api-access-r255m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:23.521506 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.521497    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-dshm" (OuterVolumeSpecName: "dshm") pod "bfdd77b6-3f47-4a14-83c1-b38e44b6e497" (UID: "bfdd77b6-3f47-4a14-83c1-b38e44b6e497"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:23.521627 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.521543    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bfdd77b6-3f47-4a14-83c1-b38e44b6e497" (UID: "bfdd77b6-3f47-4a14-83c1-b38e44b6e497"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:23.621273 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.621197    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:23.621273 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.621222    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r255m\" (UniqueName: \"kubernetes.io/projected/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-kube-api-access-r255m\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:23.621273 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.621232    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:23.621273 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.621240    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:23.621273 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.621248    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bfdd77b6-3f47-4a14-83c1-b38e44b6e497-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:23.814392 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.814358    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"]
Jul 28 17:25:23.819872 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:23.819850    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-9dmfp"]
Jul 28 17:25:24.320138 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:24.320073    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfdd77b6-3f47-4a14-83c1-b38e44b6e497" path="/var/lib/kubelet/pods/bfdd77b6-3f47-4a14-83c1-b38e44b6e497/volumes"
Jul 28 17:25:52.033048 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.033011    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"]
Jul 28 17:25:52.033538 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.033393    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfdd77b6-3f47-4a14-83c1-b38e44b6e497" containerName="main"
Jul 28 17:25:52.033538 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.033404    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfdd77b6-3f47-4a14-83c1-b38e44b6e497" containerName="main"
Jul 28 17:25:52.033670 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.033558    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfdd77b6-3f47-4a14-83c1-b38e44b6e497" containerName="main"
Jul 28 17:25:52.036599 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.036582    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.040675 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.040651    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 28 17:25:52.040815 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.040681    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 28 17:25:52.040815 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.040696    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 28 17:25:52.040922 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.040830    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"default-dockercfg-m24pw\""
Jul 28 17:25:52.046424 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.046402    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"]
Jul 28 17:25:52.170018 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.169967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.170018 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.170026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.170276 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.170128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.170276 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.170158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnk4n\" (UniqueName: \"kubernetes.io/projected/8042ab37-dc88-4d8e-ba3c-1b724723812d-kube-api-access-fnk4n\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.170276 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.170200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.170276 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.170221    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042ab37-dc88-4d8e-ba3c-1b724723812d-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.271413 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.271369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.271413 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.271416    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042ab37-dc88-4d8e-ba3c-1b724723812d-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.271610 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.271459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.271610 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.271487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.271610 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.271549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.271610 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.271572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnk4n\" (UniqueName: \"kubernetes.io/projected/8042ab37-dc88-4d8e-ba3c-1b724723812d-kube-api-access-fnk4n\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.271883 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.271857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.271995 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.271892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.271995 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.271979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.273800 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.273774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.273990 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.273975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042ab37-dc88-4d8e-ba3c-1b724723812d-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.280462 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.280441    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnk4n\" (UniqueName: \"kubernetes.io/projected/8042ab37-dc88-4d8e-ba3c-1b724723812d-kube-api-access-fnk4n\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.347936 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.347906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:52.517586 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.517562    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"]
Jul 28 17:25:52.519689 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:25:52.519660    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8042ab37_dc88_4d8e_ba3c_1b724723812d.slice/crio-a2b77fc05ed8dbff21f7d6cfe14ee20207e77514c37353fde1177c2f2ab27d63 WatchSource:0}: Error finding container a2b77fc05ed8dbff21f7d6cfe14ee20207e77514c37353fde1177c2f2ab27d63: Status 404 returned error can't find the container with id a2b77fc05ed8dbff21f7d6cfe14ee20207e77514c37353fde1177c2f2ab27d63
Jul 28 17:25:52.629015 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.628923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl" event={"ID":"8042ab37-dc88-4d8e-ba3c-1b724723812d","Type":"ContainerStarted","Data":"ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e"}
Jul 28 17:25:52.629015 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.628979    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl" event={"ID":"8042ab37-dc88-4d8e-ba3c-1b724723812d","Type":"ContainerStarted","Data":"a2b77fc05ed8dbff21f7d6cfe14ee20207e77514c37353fde1177c2f2ab27d63"}
Jul 28 17:25:52.661933 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:52.661885    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl" podStartSLOduration=0.661867419 podStartE2EDuration="661.867419ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:52.657019161 +0000 UTC m=+940.886218257" watchObservedRunningTime="2026-07-28 17:25:52.661867419 +0000 UTC m=+940.891066517"
Jul 28 17:25:53.348898 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:53.348870    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:54.826184 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:54.826153    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"]
Jul 28 17:25:55.651484 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:25:55.651443    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:25:55.651484 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:25:55.651484 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:25:55.651484 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e" cmd=["/bin/sleep","15"]
Jul 28 17:25:55.651766 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:55.651495    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl" message="" err=<
Jul 28 17:25:55.651766 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:25:55.651766 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:25:55.651766 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:25:55.651766 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:25:55.651510    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:25:55.651766 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:25:55.651766 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:25:55.651766 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl" podUID="8042ab37-dc88-4d8e-ba3c-1b724723812d" containerName="main" containerID="cri-o://ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e"
Jul 28 17:25:55.651766 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:55.651551    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl" podUID="8042ab37-dc88-4d8e-ba3c-1b724723812d" containerName="main" containerID="cri-o://ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e" gracePeriod=60
Jul 28 17:25:56.001957 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.001934    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:56.102876 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.102842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-model-cache\") pod \"8042ab37-dc88-4d8e-ba3c-1b724723812d\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") "
Jul 28 17:25:56.103044 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.102921    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-home\") pod \"8042ab37-dc88-4d8e-ba3c-1b724723812d\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") "
Jul 28 17:25:56.103044 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.102947    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042ab37-dc88-4d8e-ba3c-1b724723812d-tls-certs\") pod \"8042ab37-dc88-4d8e-ba3c-1b724723812d\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") "
Jul 28 17:25:56.103044 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.102995    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-dshm\") pod \"8042ab37-dc88-4d8e-ba3c-1b724723812d\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") "
Jul 28 17:25:56.103044 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.103023    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-tmp-dir\") pod \"8042ab37-dc88-4d8e-ba3c-1b724723812d\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") "
Jul 28 17:25:56.103278 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.103047    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnk4n\" (UniqueName: \"kubernetes.io/projected/8042ab37-dc88-4d8e-ba3c-1b724723812d-kube-api-access-fnk4n\") pod \"8042ab37-dc88-4d8e-ba3c-1b724723812d\" (UID: \"8042ab37-dc88-4d8e-ba3c-1b724723812d\") "
Jul 28 17:25:56.103278 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.103187    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-home" (OuterVolumeSpecName: "home") pod "8042ab37-dc88-4d8e-ba3c-1b724723812d" (UID: "8042ab37-dc88-4d8e-ba3c-1b724723812d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:56.103278 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.103208    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-model-cache" (OuterVolumeSpecName: "model-cache") pod "8042ab37-dc88-4d8e-ba3c-1b724723812d" (UID: "8042ab37-dc88-4d8e-ba3c-1b724723812d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:56.103440 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.103334    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8042ab37-dc88-4d8e-ba3c-1b724723812d" (UID: "8042ab37-dc88-4d8e-ba3c-1b724723812d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:56.103440 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.103416    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:56.103440 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.103432    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:56.103440 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.103440    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:56.105289 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.105268    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-dshm" (OuterVolumeSpecName: "dshm") pod "8042ab37-dc88-4d8e-ba3c-1b724723812d" (UID: "8042ab37-dc88-4d8e-ba3c-1b724723812d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:25:56.105375 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.105285    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8042ab37-dc88-4d8e-ba3c-1b724723812d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8042ab37-dc88-4d8e-ba3c-1b724723812d" (UID: "8042ab37-dc88-4d8e-ba3c-1b724723812d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:25:56.105375 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.105296    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8042ab37-dc88-4d8e-ba3c-1b724723812d-kube-api-access-fnk4n" (OuterVolumeSpecName: "kube-api-access-fnk4n") pod "8042ab37-dc88-4d8e-ba3c-1b724723812d" (UID: "8042ab37-dc88-4d8e-ba3c-1b724723812d"). InnerVolumeSpecName "kube-api-access-fnk4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:25:56.203855 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.203777    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8042ab37-dc88-4d8e-ba3c-1b724723812d-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:56.203855 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.203805    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8042ab37-dc88-4d8e-ba3c-1b724723812d-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:56.203855 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.203815    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnk4n\" (UniqueName: \"kubernetes.io/projected/8042ab37-dc88-4d8e-ba3c-1b724723812d-kube-api-access-fnk4n\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:25:56.648166 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.648135    2579 generic.go:356] "Generic (PLEG): container finished" podID="8042ab37-dc88-4d8e-ba3c-1b724723812d" containerID="ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e" exitCode=0
Jul 28 17:25:56.648331 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.648177    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"
Jul 28 17:25:56.648331 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.648195    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl" event={"ID":"8042ab37-dc88-4d8e-ba3c-1b724723812d","Type":"ContainerDied","Data":"ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e"}
Jul 28 17:25:56.648331 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.648238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl" event={"ID":"8042ab37-dc88-4d8e-ba3c-1b724723812d","Type":"ContainerDied","Data":"a2b77fc05ed8dbff21f7d6cfe14ee20207e77514c37353fde1177c2f2ab27d63"}
Jul 28 17:25:56.648331 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.648261    2579 scope.go:117] "RemoveContainer" containerID="ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e"
Jul 28 17:25:56.657055 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.656888    2579 scope.go:117] "RemoveContainer" containerID="ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e"
Jul 28 17:25:56.657173 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:25:56.657145    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e\": container with ID starting with ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e not found: ID does not exist" containerID="ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e"
Jul 28 17:25:56.657233 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.657187    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e"} err="failed to get container status \"ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e\": rpc error: code = NotFound desc = could not find container \"ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e\": container with ID starting with ce37915525141a9cee020a482188e23d06e54572f890e55663122c60d838fb3e not found: ID does not exist"
Jul 28 17:25:56.664751 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.664730    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"]
Jul 28 17:25:56.668286 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:56.668267    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/gw-section-name-router-with-gat-2f0a622e-kserve-8dc8c8494-424jl"]
Jul 28 17:25:58.319984 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:25:58.319950    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8042ab37-dc88-4d8e-ba3c-1b724723812d" path="/var/lib/kubelet/pods/8042ab37-dc88-4d8e-ba3c-1b724723812d/volumes"
Jul 28 17:26:11.747446 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.747414    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"]
Jul 28 17:26:11.747958 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.747939    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8042ab37-dc88-4d8e-ba3c-1b724723812d" containerName="main"
Jul 28 17:26:11.748036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.747961    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8042ab37-dc88-4d8e-ba3c-1b724723812d" containerName="main"
Jul 28 17:26:11.748122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.748101    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8042ab37-dc88-4d8e-ba3c-1b724723812d" containerName="main"
Jul 28 17:26:11.750946 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.750925    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.753570 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.753551    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 28 17:26:11.754813 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.754793    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-6rtnl\""
Jul 28 17:26:11.754892 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.754828    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 28 17:26:11.754892 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.754803    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 28 17:26:11.762420 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.762399    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"]
Jul 28 17:26:11.836619 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.836590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4f5w\" (UniqueName: \"kubernetes.io/projected/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-kube-api-access-g4f5w\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.836757 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.836641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.836757 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.836716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.836757 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.836750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.836868 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.836767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.836868 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.836820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.937548 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.937513    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.937714 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.937569    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.937714 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.937600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.937714 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.937621    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.937714 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.937667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.937714 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.937704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4f5w\" (UniqueName: \"kubernetes.io/projected/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-kube-api-access-g4f5w\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.938128 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.938067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.938249 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.938155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.938249 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.938174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.939880 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.939856    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.940022 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.940006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:11.946009 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:11.945991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4f5w\" (UniqueName: \"kubernetes.io/projected/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-kube-api-access-g4f5w\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:12.061309 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:12.061241    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:12.194860 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:12.194834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"]
Jul 28 17:26:12.196826 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:26:12.196800    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ad7167_6500_41a3_8b5b_e55b8b77fec4.slice/crio-57b4bcb1981ea65b267d85aa2a045287f766827f7af53a95f5a395ac56e4d103 WatchSource:0}: Error finding container 57b4bcb1981ea65b267d85aa2a045287f766827f7af53a95f5a395ac56e4d103: Status 404 returned error can't find the container with id 57b4bcb1981ea65b267d85aa2a045287f766827f7af53a95f5a395ac56e4d103
Jul 28 17:26:12.253829 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:12.253807    2579 scope.go:117] "RemoveContainer" containerID="7c04719acc0a6de1bff70ef3955315b977344bc09ea1eedb8d44360374d15f2d"
Jul 28 17:26:12.726287 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:12.726250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68" event={"ID":"a5ad7167-6500-41a3-8b5b-e55b8b77fec4","Type":"ContainerStarted","Data":"7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd"}
Jul 28 17:26:12.726447 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:12.726303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68" event={"ID":"a5ad7167-6500-41a3-8b5b-e55b8b77fec4","Type":"ContainerStarted","Data":"57b4bcb1981ea65b267d85aa2a045287f766827f7af53a95f5a395ac56e4d103"}
Jul 28 17:26:12.747563 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:12.747517    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68" podStartSLOduration=1.747501908 podStartE2EDuration="1.747501908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:12.746215125 +0000 UTC m=+960.975414219" watchObservedRunningTime="2026-07-28 17:26:12.747501908 +0000 UTC m=+960.976701003"
Jul 28 17:26:13.062161 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:13.062066    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:14.680905 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:14.680870    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"]
Jul 28 17:26:14.864590 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:26:14.864548    2579 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 28 17:26:14.864801 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:26:14.864626    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tls-certs podName:a5ad7167-6500-41a3-8b5b-e55b8b77fec4 nodeName:}" failed. No retries permitted until 2026-07-28 17:26:15.364610939 +0000 UTC m=+963.593810018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68" (UID: "a5ad7167-6500-41a3-8b5b-e55b8b77fec4") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 28 17:26:15.368937 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:26:15.368904    2579 secret.go:189] Couldn't get secret e2e-test-gateway-section-name-propagation-94799d44/gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs: secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 28 17:26:15.369134 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:26:15.368966    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tls-certs podName:a5ad7167-6500-41a3-8b5b-e55b8b77fec4 nodeName:}" failed. No retries permitted until 2026-07-28 17:26:16.368953366 +0000 UTC m=+964.598152440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tls-certs") pod "gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68" (UID: "a5ad7167-6500-41a3-8b5b-e55b8b77fec4") : secret "gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs" not found
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:26:15.748519    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd" cmd=["/bin/sleep","15"]
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:15.748569    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68" message="" err=<
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:26:15.748581    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:26:15.748593 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68" podUID="a5ad7167-6500-41a3-8b5b-e55b8b77fec4" containerName="main" containerID="cri-o://7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd"
Jul 28 17:26:15.749290 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:15.748630    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68" podUID="a5ad7167-6500-41a3-8b5b-e55b8b77fec4" containerName="main" containerID="cri-o://7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd" gracePeriod=60
Jul 28 17:26:16.097830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.097805    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:16.177492 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177465    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-home\") pod \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") "
Jul 28 17:26:16.177627 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-model-cache\") pod \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") "
Jul 28 17:26:16.177627 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177549    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tmp-dir\") pod \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") "
Jul 28 17:26:16.177627 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177571    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tls-certs\") pod \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") "
Jul 28 17:26:16.177627 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177597    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4f5w\" (UniqueName: \"kubernetes.io/projected/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-kube-api-access-g4f5w\") pod \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") "
Jul 28 17:26:16.177627 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177619    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-dshm\") pod \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\" (UID: \"a5ad7167-6500-41a3-8b5b-e55b8b77fec4\") "
Jul 28 17:26:16.177888 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177712    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-home" (OuterVolumeSpecName: "home") pod "a5ad7167-6500-41a3-8b5b-e55b8b77fec4" (UID: "a5ad7167-6500-41a3-8b5b-e55b8b77fec4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:16.177888 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-model-cache" (OuterVolumeSpecName: "model-cache") pod "a5ad7167-6500-41a3-8b5b-e55b8b77fec4" (UID: "a5ad7167-6500-41a3-8b5b-e55b8b77fec4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:16.177888 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177856    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a5ad7167-6500-41a3-8b5b-e55b8b77fec4" (UID: "a5ad7167-6500-41a3-8b5b-e55b8b77fec4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:16.178011 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177918    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:26:16.178011 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177935    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:26:16.178011 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.177947    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:26:16.179841 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.179812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-dshm" (OuterVolumeSpecName: "dshm") pod "a5ad7167-6500-41a3-8b5b-e55b8b77fec4" (UID: "a5ad7167-6500-41a3-8b5b-e55b8b77fec4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:16.179939 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.179861    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a5ad7167-6500-41a3-8b5b-e55b8b77fec4" (UID: "a5ad7167-6500-41a3-8b5b-e55b8b77fec4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:26:16.179939 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.179897    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-kube-api-access-g4f5w" (OuterVolumeSpecName: "kube-api-access-g4f5w") pod "a5ad7167-6500-41a3-8b5b-e55b8b77fec4" (UID: "a5ad7167-6500-41a3-8b5b-e55b8b77fec4"). InnerVolumeSpecName "kube-api-access-g4f5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:16.279216 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.279160    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:26:16.279216 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.279180    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4f5w\" (UniqueName: \"kubernetes.io/projected/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-kube-api-access-g4f5w\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:26:16.279216 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.279189    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5ad7167-6500-41a3-8b5b-e55b8b77fec4-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:26:16.745543 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.745506    2579 generic.go:356] "Generic (PLEG): container finished" podID="a5ad7167-6500-41a3-8b5b-e55b8b77fec4" containerID="7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd" exitCode=0
Jul 28 17:26:16.745709 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.745546    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"
Jul 28 17:26:16.745709 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.745553    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68" event={"ID":"a5ad7167-6500-41a3-8b5b-e55b8b77fec4","Type":"ContainerDied","Data":"7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd"}
Jul 28 17:26:16.745709 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.745603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68" event={"ID":"a5ad7167-6500-41a3-8b5b-e55b8b77fec4","Type":"ContainerDied","Data":"57b4bcb1981ea65b267d85aa2a045287f766827f7af53a95f5a395ac56e4d103"}
Jul 28 17:26:16.745709 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.745637    2579 scope.go:117] "RemoveContainer" containerID="7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd"
Jul 28 17:26:16.754490 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.754259    2579 scope.go:117] "RemoveContainer" containerID="7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd"
Jul 28 17:26:16.754710 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:26:16.754530    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd\": container with ID starting with 7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd not found: ID does not exist" containerID="7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd"
Jul 28 17:26:16.754710 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.754551    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd"} err="failed to get container status \"7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd\": rpc error: code = NotFound desc = could not find container \"7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd\": container with ID starting with 7fffd494ca3b220b63c44d5b41de748070a6b9f7e03a8e29fdcf62b18cad90cd not found: ID does not exist"
Jul 28 17:26:16.762158 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.762136    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"]
Jul 28 17:26:16.767620 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:16.767600    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbfgjm68"]
Jul 28 17:26:18.320073 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:18.320039    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ad7167-6500-41a3-8b5b-e55b8b77fec4" path="/var/lib/kubelet/pods/a5ad7167-6500-41a3-8b5b-e55b8b77fec4/volumes"
Jul 28 17:26:35.438711 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.438632    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"]
Jul 28 17:26:35.439157 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.438992    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ad7167-6500-41a3-8b5b-e55b8b77fec4" containerName="main"
Jul 28 17:26:35.439157 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.439002    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ad7167-6500-41a3-8b5b-e55b8b77fec4" containerName="main"
Jul 28 17:26:35.439157 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.439096    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ad7167-6500-41a3-8b5b-e55b8b77fec4" containerName="main"
Jul 28 17:26:35.443567 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.443548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.446589 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.446562    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Jul 28 17:26:35.446702 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.446612    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 28 17:26:35.446702 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.446579    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 28 17:26:35.446829 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.446762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-29hj4\""
Jul 28 17:26:35.447034 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.447015    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 28 17:26:35.454884 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.454865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"]
Jul 28 17:26:35.550302 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.550269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-home\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.550302 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.550305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56h7k\" (UniqueName: \"kubernetes.io/projected/de0eee21-b646-4943-a7e1-c262e726c15a-kube-api-access-56h7k\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.550528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.550333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-kserve-provision-location\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.550528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.550360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-model-cache\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.550528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.550386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-dshm\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.550528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.550412    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-tmp-dir\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.550528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.550465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de0eee21-b646-4943-a7e1-c262e726c15a-tls-certs\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.651428 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.651391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-home\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.651601 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.651443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56h7k\" (UniqueName: \"kubernetes.io/projected/de0eee21-b646-4943-a7e1-c262e726c15a-kube-api-access-56h7k\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.651601 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.651481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-kserve-provision-location\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.651601 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.651508    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-model-cache\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.651601 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.651549    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-dshm\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.651601 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.651593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-tmp-dir\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.651855 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.651626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de0eee21-b646-4943-a7e1-c262e726c15a-tls-certs\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.651855 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.651833    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-home\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.651965 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.651928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-model-cache\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.652018 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.651988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-tmp-dir\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.652213 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.652192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-kserve-provision-location\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.654030 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.654004    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de0eee21-b646-4943-a7e1-c262e726c15a-tls-certs\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.654166 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.654113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-dshm\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.660216 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.660195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56h7k\" (UniqueName: \"kubernetes.io/projected/de0eee21-b646-4943-a7e1-c262e726c15a-kube-api-access-56h7k\") pod \"auth-enabled-test-kserve-777b6c898c-k7wpb\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.755545 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.755469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:26:35.893927 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:35.893903    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"]
Jul 28 17:26:35.896202 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:26:35.896174    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0eee21_b646_4943_a7e1_c262e726c15a.slice/crio-c2cbbdaabf054813c5eff7726674606b76c4053f22d43b8073f383a304884d0a WatchSource:0}: Error finding container c2cbbdaabf054813c5eff7726674606b76c4053f22d43b8073f383a304884d0a: Status 404 returned error can't find the container with id c2cbbdaabf054813c5eff7726674606b76c4053f22d43b8073f383a304884d0a
Jul 28 17:26:36.846917 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:36.846869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" event={"ID":"de0eee21-b646-4943-a7e1-c262e726c15a","Type":"ContainerStarted","Data":"c2cbbdaabf054813c5eff7726674606b76c4053f22d43b8073f383a304884d0a"}
Jul 28 17:26:39.864767 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:26:39.864732    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" event={"ID":"de0eee21-b646-4943-a7e1-c262e726c15a","Type":"ContainerStarted","Data":"1a968eb11364abc50b0762f448ccddc6c0d72472aac7906f8be1f290b4996bfa"}
Jul 28 17:27:08.969352 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:08.969314    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 17:27:08.973486 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:08.973465    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:08.976285 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:08.976256    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 28 17:27:08.976373 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:08.976279    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 28 17:27:08.977481 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:08.977454    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-lr6wf\""
Jul 28 17:27:08.977481 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:08.977458    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 28 17:27:08.977658 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:08.977519    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-fkhh5\""
Jul 28 17:27:08.977658 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:08.977519    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 28 17:27:08.982603 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:08.982582    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 17:27:09.046449 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.046417    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.046449 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.046458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk6dn\" (UniqueName: \"kubernetes.io/projected/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kube-api-access-rk6dn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.046673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.046500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.046673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.046557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.046673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.046626    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.046673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.046651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.046673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.046669    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.147840 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.147799    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.147840 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.147841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk6dn\" (UniqueName: \"kubernetes.io/projected/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kube-api-access-rk6dn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.148188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.147879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.148188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.147915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.148188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.147951    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.148188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.147974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.148188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.148000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.148461 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.148238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.148461 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.148329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.148461 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.148385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.148461 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.148413    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.150176 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.150154    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.150274 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.150252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.155362 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.155338    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk6dn\" (UniqueName: \"kubernetes.io/projected/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kube-api-access-rk6dn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.285444 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.285356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:27:09.447812 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:09.447777    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 17:27:09.452127 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:27:09.452044    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2809b9c7_f6f7_4960_b623_c0f4872f88c1.slice/crio-0fdbe16c109ce46b4fb675f214356d87b7e30b7a85c7d6b880ecb664ba8cccdb WatchSource:0}: Error finding container 0fdbe16c109ce46b4fb675f214356d87b7e30b7a85c7d6b880ecb664ba8cccdb: Status 404 returned error can't find the container with id 0fdbe16c109ce46b4fb675f214356d87b7e30b7a85c7d6b880ecb664ba8cccdb
Jul 28 17:27:10.019060 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:10.019021    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2809b9c7-f6f7-4960-b623-c0f4872f88c1","Type":"ContainerStarted","Data":"256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697"}
Jul 28 17:27:10.019474 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:10.019092    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2809b9c7-f6f7-4960-b623-c0f4872f88c1","Type":"ContainerStarted","Data":"0fdbe16c109ce46b4fb675f214356d87b7e30b7a85c7d6b880ecb664ba8cccdb"}
Jul 28 17:27:14.041698 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:14.041663    2579 generic.go:356] "Generic (PLEG): container finished" podID="2809b9c7-f6f7-4960-b623-c0f4872f88c1" containerID="256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697" exitCode=0
Jul 28 17:27:14.042148 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:14.041736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2809b9c7-f6f7-4960-b623-c0f4872f88c1","Type":"ContainerDied","Data":"256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697"}
Jul 28 17:27:45.231135 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:45.231064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2809b9c7-f6f7-4960-b623-c0f4872f88c1","Type":"ContainerStarted","Data":"85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df"}
Jul 28 17:27:45.254611 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:45.254564    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=3.327498003 podStartE2EDuration="37.254548871s" totalImagesPullingTime="30.170104666s" totalInitContainerRuntime="3.756946202s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:45.252793432 +0000 UTC m=+1053.481992530" watchObservedRunningTime="2026-07-28 17:27:45.254548871 +0000 UTC m=+1053.483747966"
Jul 28 17:27:51.266261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:51.266219    2579 generic.go:356] "Generic (PLEG): container finished" podID="de0eee21-b646-4943-a7e1-c262e726c15a" containerID="1a968eb11364abc50b0762f448ccddc6c0d72472aac7906f8be1f290b4996bfa" exitCode=0
Jul 28 17:27:51.266698 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:51.266287    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" event={"ID":"de0eee21-b646-4943-a7e1-c262e726c15a","Type":"ContainerDied","Data":"1a968eb11364abc50b0762f448ccddc6c0d72472aac7906f8be1f290b4996bfa"}
Jul 28 17:27:52.275381 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:52.275342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" event={"ID":"de0eee21-b646-4943-a7e1-c262e726c15a","Type":"ContainerStarted","Data":"4cd40440d24ffab31ee779b77d9d62628debd3fa6d838db7684d22531aaa9f43"}
Jul 28 17:27:52.303904 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:52.303850    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podStartSLOduration=2.458077998 podStartE2EDuration="1m17.3038345s" totalImagesPullingTime="3.504848992s" totalInitContainerRuntime="1m11.34090751s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:52.302233251 +0000 UTC m=+1060.531432373" watchObservedRunningTime="2026-07-28 17:27:52.3038345 +0000 UTC m=+1060.533033596"
Jul 28 17:27:52.756204 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:52.756164    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:27:55.756149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:55.756109    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:27:55.757942 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:27:55.757904    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:28:05.756837 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:28:05.756722    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:28:15.756798 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:28:15.756744    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:28:25.756526 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:28:25.756469    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:28:35.756793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:28:35.756741    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:28:45.756633 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:28:45.756593    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:28:55.756206 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:28:55.756158    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:29:05.755990 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:29:05.755935    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:29:15.756330 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:29:15.756289    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:29:25.756320 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:29:25.756268    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:29:35.756252 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:29:35.756165    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:29:45.756554 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:29:45.756511    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:29:55.756423 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:29:55.756378    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:30:05.766184 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:05.766152    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:30:05.773725 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:05.773696    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:30:10.453951 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:10.453908    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"]
Jul 28 17:30:12.260646 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:12.260616    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:30:12.268846 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:12.268819    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:30:20.314809 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:20.314770    2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-lr6wf\" not found"
Jul 28 17:30:20.448784 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:20.448754    2579 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:20.448945 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:20.448822    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs podName:2809b9c7-f6f7-4960-b623-c0f4872f88c1 nodeName:}" failed. No retries permitted until 2026-07-28 17:30:20.948805529 +0000 UTC m=+1209.178004603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:20.953714 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:20.953678    2579 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:20.953888 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:20.953757    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs podName:2809b9c7-f6f7-4960-b623-c0f4872f88c1 nodeName:}" failed. No retries permitted until 2026-07-28 17:30:21.95374146 +0000 UTC m=+1210.182940534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:21.962985 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:21.962953    2579 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:21.963392 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:21.963019    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs podName:2809b9c7-f6f7-4960-b623-c0f4872f88c1 nodeName:}" failed. No retries permitted until 2026-07-28 17:30:23.963006066 +0000 UTC m=+1212.192205139 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:23.980625 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:23.980590    2579 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:23.981019 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:23.980669    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs podName:2809b9c7-f6f7-4960-b623-c0f4872f88c1 nodeName:}" failed. No retries permitted until 2026-07-28 17:30:27.980653636 +0000 UTC m=+1216.209852711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:25.478929 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:25.478894    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" containerID="cri-o://4cd40440d24ffab31ee779b77d9d62628debd3fa6d838db7684d22531aaa9f43" gracePeriod=45
Jul 28 17:30:25.767138 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:25.767034    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.36:8000/health\": dial tcp 10.132.0.36:8000: connect: connection refused"
Jul 28 17:30:26.111025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.110993    2579 generic.go:356] "Generic (PLEG): container finished" podID="de0eee21-b646-4943-a7e1-c262e726c15a" containerID="4cd40440d24ffab31ee779b77d9d62628debd3fa6d838db7684d22531aaa9f43" exitCode=0
Jul 28 17:30:26.111193 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.111067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" event={"ID":"de0eee21-b646-4943-a7e1-c262e726c15a","Type":"ContainerDied","Data":"4cd40440d24ffab31ee779b77d9d62628debd3fa6d838db7684d22531aaa9f43"}
Jul 28 17:30:26.155888 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.155866    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:30:26.302395 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.302361    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-dshm\") pod \"de0eee21-b646-4943-a7e1-c262e726c15a\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") "
Jul 28 17:30:26.302568 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.302405    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de0eee21-b646-4943-a7e1-c262e726c15a-tls-certs\") pod \"de0eee21-b646-4943-a7e1-c262e726c15a\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") "
Jul 28 17:30:26.302568 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.302436    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-model-cache\") pod \"de0eee21-b646-4943-a7e1-c262e726c15a\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") "
Jul 28 17:30:26.302568 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.302468    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-kserve-provision-location\") pod \"de0eee21-b646-4943-a7e1-c262e726c15a\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") "
Jul 28 17:30:26.302741 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.302632    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-home\") pod \"de0eee21-b646-4943-a7e1-c262e726c15a\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") "
Jul 28 17:30:26.302741 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.302672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56h7k\" (UniqueName: \"kubernetes.io/projected/de0eee21-b646-4943-a7e1-c262e726c15a-kube-api-access-56h7k\") pod \"de0eee21-b646-4943-a7e1-c262e726c15a\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") "
Jul 28 17:30:26.302741 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.302719    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-model-cache" (OuterVolumeSpecName: "model-cache") pod "de0eee21-b646-4943-a7e1-c262e726c15a" (UID: "de0eee21-b646-4943-a7e1-c262e726c15a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:26.302741 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.302733    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-tmp-dir\") pod \"de0eee21-b646-4943-a7e1-c262e726c15a\" (UID: \"de0eee21-b646-4943-a7e1-c262e726c15a\") "
Jul 28 17:30:26.303906 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.303823    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:26.306105 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.304594    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-home" (OuterVolumeSpecName: "home") pod "de0eee21-b646-4943-a7e1-c262e726c15a" (UID: "de0eee21-b646-4943-a7e1-c262e726c15a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:26.315694 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.315667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-dshm" (OuterVolumeSpecName: "dshm") pod "de0eee21-b646-4943-a7e1-c262e726c15a" (UID: "de0eee21-b646-4943-a7e1-c262e726c15a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:26.316345 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.316296    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0eee21-b646-4943-a7e1-c262e726c15a-kube-api-access-56h7k" (OuterVolumeSpecName: "kube-api-access-56h7k") pod "de0eee21-b646-4943-a7e1-c262e726c15a" (UID: "de0eee21-b646-4943-a7e1-c262e726c15a"). InnerVolumeSpecName "kube-api-access-56h7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:26.317678 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.317649    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de0eee21-b646-4943-a7e1-c262e726c15a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "de0eee21-b646-4943-a7e1-c262e726c15a" (UID: "de0eee21-b646-4943-a7e1-c262e726c15a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:26.320059 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.320033    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "de0eee21-b646-4943-a7e1-c262e726c15a" (UID: "de0eee21-b646-4943-a7e1-c262e726c15a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:26.367753 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.367717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "de0eee21-b646-4943-a7e1-c262e726c15a" (UID: "de0eee21-b646-4943-a7e1-c262e726c15a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:26.404729 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.404704    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:26.404729 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.404728    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:26.404878 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.404739    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56h7k\" (UniqueName: \"kubernetes.io/projected/de0eee21-b646-4943-a7e1-c262e726c15a-kube-api-access-56h7k\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:26.404878 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.404748    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:26.404878 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.404758    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/de0eee21-b646-4943-a7e1-c262e726c15a-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:26.404878 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:26.404765    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de0eee21-b646-4943-a7e1-c262e726c15a-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:27.117804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:27.117776    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"
Jul 28 17:30:27.118224 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:27.117813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb" event={"ID":"de0eee21-b646-4943-a7e1-c262e726c15a","Type":"ContainerDied","Data":"c2cbbdaabf054813c5eff7726674606b76c4053f22d43b8073f383a304884d0a"}
Jul 28 17:30:27.118224 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:27.117885    2579 scope.go:117] "RemoveContainer" containerID="4cd40440d24ffab31ee779b77d9d62628debd3fa6d838db7684d22531aaa9f43"
Jul 28 17:30:27.127452 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:27.127433    2579 scope.go:117] "RemoveContainer" containerID="1a968eb11364abc50b0762f448ccddc6c0d72472aac7906f8be1f290b4996bfa"
Jul 28 17:30:27.142760 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:27.142737    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"]
Jul 28 17:30:27.148239 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:27.148219    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-777b6c898c-k7wpb"]
Jul 28 17:30:28.020495 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:28.020458    2579 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:28.020679 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:28.020536    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs podName:2809b9c7-f6f7-4960-b623-c0f4872f88c1 nodeName:}" failed. No retries permitted until 2026-07-28 17:30:36.020521436 +0000 UTC m=+1224.249720509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:28.319792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:28.319759    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" path="/var/lib/kubelet/pods/de0eee21-b646-4943-a7e1-c262e726c15a/volumes"
Jul 28 17:30:32.837553 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:32.837523    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 17:30:36.092638 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:36.092603    2579 secret.go:189] Couldn't get secret e2e-test-llm-inference-service-63a03ef4/llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs: secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:36.093042 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:36.092673    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs podName:2809b9c7-f6f7-4960-b623-c0f4872f88c1 nodeName:}" failed. No retries permitted until 2026-07-28 17:30:52.092657112 +0000 UTC m=+1240.321856188 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs") pod "llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1") : secret "llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs" not found
Jul 28 17:30:47.859425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:47.859385    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="2809b9c7-f6f7-4960-b623-c0f4872f88c1" containerName="main" containerID="cri-o://85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df" gracePeriod=45
Jul 28 17:30:48.609942 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.609919    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:30:48.710816 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.710749    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tmp-dir\") pod \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") "
Jul 28 17:30:48.710816 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.710780    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-model-cache\") pod \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") "
Jul 28 17:30:48.710816 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.710809    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kserve-provision-location\") pod \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") "
Jul 28 17:30:48.711045 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.710851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-home\") pod \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") "
Jul 28 17:30:48.711045 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.710907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs\") pod \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") "
Jul 28 17:30:48.711045 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.710933    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk6dn\" (UniqueName: \"kubernetes.io/projected/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kube-api-access-rk6dn\") pod \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") "
Jul 28 17:30:48.711045 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.710976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-dshm\") pod \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\" (UID: \"2809b9c7-f6f7-4960-b623-c0f4872f88c1\") "
Jul 28 17:30:48.711294 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.711047    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "2809b9c7-f6f7-4960-b623-c0f4872f88c1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:48.711294 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.711269    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:48.711518 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.711491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-home" (OuterVolumeSpecName: "home") pod "2809b9c7-f6f7-4960-b623-c0f4872f88c1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:48.713288 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.713262    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kube-api-access-rk6dn" (OuterVolumeSpecName: "kube-api-access-rk6dn") pod "2809b9c7-f6f7-4960-b623-c0f4872f88c1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1"). InnerVolumeSpecName "kube-api-access-rk6dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:48.713400 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.713356    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-dshm" (OuterVolumeSpecName: "dshm") pod "2809b9c7-f6f7-4960-b623-c0f4872f88c1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:48.713400 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.713378    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2809b9c7-f6f7-4960-b623-c0f4872f88c1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:48.723132 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.723063    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2809b9c7-f6f7-4960-b623-c0f4872f88c1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:48.766212 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.766188    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2809b9c7-f6f7-4960-b623-c0f4872f88c1" (UID: "2809b9c7-f6f7-4960-b623-c0f4872f88c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:48.812404 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.812385    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:48.812404 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.812402    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:48.812523 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.812413    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk6dn\" (UniqueName: \"kubernetes.io/projected/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kube-api-access-rk6dn\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:48.812523 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.812421    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:48.812523 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.812430    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:48.812523 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:48.812437    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2809b9c7-f6f7-4960-b623-c0f4872f88c1-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:30:49.227530 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.227499    2579 generic.go:356] "Generic (PLEG): container finished" podID="2809b9c7-f6f7-4960-b623-c0f4872f88c1" containerID="85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df" exitCode=0
Jul 28 17:30:49.228010 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.227560    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 28 17:30:49.228010 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.227583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2809b9c7-f6f7-4960-b623-c0f4872f88c1","Type":"ContainerDied","Data":"85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df"}
Jul 28 17:30:49.228010 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.227637    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"2809b9c7-f6f7-4960-b623-c0f4872f88c1","Type":"ContainerDied","Data":"0fdbe16c109ce46b4fb675f214356d87b7e30b7a85c7d6b880ecb664ba8cccdb"}
Jul 28 17:30:49.228010 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.227661    2579 scope.go:117] "RemoveContainer" containerID="85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df"
Jul 28 17:30:49.237436 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.237420    2579 scope.go:117] "RemoveContainer" containerID="256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697"
Jul 28 17:30:49.245353 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.245333    2579 scope.go:117] "RemoveContainer" containerID="85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df"
Jul 28 17:30:49.245591 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:49.245570    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df\": container with ID starting with 85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df not found: ID does not exist" containerID="85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df"
Jul 28 17:30:49.245638 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.245600    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df"} err="failed to get container status \"85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df\": rpc error: code = NotFound desc = could not find container \"85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df\": container with ID starting with 85884bb04ecca472fd2e63b430af26108d7ea3c0f61f2be92393c21b7ed762df not found: ID does not exist"
Jul 28 17:30:49.245638 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.245616    2579 scope.go:117] "RemoveContainer" containerID="256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697"
Jul 28 17:30:49.245820 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:30:49.245803    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697\": container with ID starting with 256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697 not found: ID does not exist" containerID="256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697"
Jul 28 17:30:49.245869 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.245827    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697"} err="failed to get container status \"256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697\": rpc error: code = NotFound desc = could not find container \"256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697\": container with ID starting with 256c0afa7d3b6879a72b74affb3ddb48764b648971a3e2856fdb3f439d2d9697 not found: ID does not exist"
Jul 28 17:30:49.259149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.259119    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 17:30:49.260572 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:49.260556    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 28 17:30:50.318951 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:30:50.318922    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2809b9c7-f6f7-4960-b623-c0f4872f88c1" path="/var/lib/kubelet/pods/2809b9c7-f6f7-4960-b623-c0f4872f88c1/volumes"
Jul 28 17:31:14.755568 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.755538    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"]
Jul 28 17:31:14.755960 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.755903    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2809b9c7-f6f7-4960-b623-c0f4872f88c1" containerName="storage-initializer"
Jul 28 17:31:14.755960 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.755913    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2809b9c7-f6f7-4960-b623-c0f4872f88c1" containerName="storage-initializer"
Jul 28 17:31:14.755960 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.755924    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="storage-initializer"
Jul 28 17:31:14.755960 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.755930    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="storage-initializer"
Jul 28 17:31:14.755960 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.755936    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main"
Jul 28 17:31:14.755960 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.755941    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main"
Jul 28 17:31:14.755960 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.755949    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2809b9c7-f6f7-4960-b623-c0f4872f88c1" containerName="main"
Jul 28 17:31:14.755960 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.755954    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2809b9c7-f6f7-4960-b623-c0f4872f88c1" containerName="main"
Jul 28 17:31:14.756225 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.756009    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0eee21-b646-4943-a7e1-c262e726c15a" containerName="main"
Jul 28 17:31:14.756225 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.756019    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2809b9c7-f6f7-4960-b623-c0f4872f88c1" containerName="main"
Jul 28 17:31:14.760251 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.760235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:14.764416 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.764397    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 28 17:31:14.764603 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.764542    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 28 17:31:14.764721 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.764552    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-lfd5h\""
Jul 28 17:31:14.764721 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.764618    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 28 17:31:14.771498 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.771476    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"]
Jul 28 17:31:14.943735 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.943701    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:14.943735 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.943735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:14.943971 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.943753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nsxf\" (UniqueName: \"kubernetes.io/projected/7764fe92-58fa-4a24-874a-9ec3419eca1d-kube-api-access-8nsxf\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:14.943971 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.943819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:14.943971 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.943876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:14.944116 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:14.943974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7764fe92-58fa-4a24-874a-9ec3419eca1d-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.045017 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.044930    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.045017 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.044974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.045276 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.045033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7764fe92-58fa-4a24-874a-9ec3419eca1d-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.045276 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.045071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.045276 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.045123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.045276 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.045146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nsxf\" (UniqueName: \"kubernetes.io/projected/7764fe92-58fa-4a24-874a-9ec3419eca1d-kube-api-access-8nsxf\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.045484 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.045370    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.045484 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.045476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.045571 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.045500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.047335 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.047307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.047644 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.047613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7764fe92-58fa-4a24-874a-9ec3419eca1d-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.054353 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.054334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nsxf\" (UniqueName: \"kubernetes.io/projected/7764fe92-58fa-4a24-874a-9ec3419eca1d-kube-api-access-8nsxf\") pod \"scheduler-inline-config-test-kserve-8655bc757-j5dwd\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.071116 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.071093    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:15.209474 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.209449    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"]
Jul 28 17:31:15.211471 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:31:15.211444    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7764fe92_58fa_4a24_874a_9ec3419eca1d.slice/crio-4b69b252b0b9830cbcbe6f6c9e409311eec30b2644029a42237ba45d14abe311 WatchSource:0}: Error finding container 4b69b252b0b9830cbcbe6f6c9e409311eec30b2644029a42237ba45d14abe311: Status 404 returned error can't find the container with id 4b69b252b0b9830cbcbe6f6c9e409311eec30b2644029a42237ba45d14abe311
Jul 28 17:31:15.352033 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.351999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd" event={"ID":"7764fe92-58fa-4a24-874a-9ec3419eca1d","Type":"ContainerStarted","Data":"6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59"}
Jul 28 17:31:15.352241 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.352052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd" event={"ID":"7764fe92-58fa-4a24-874a-9ec3419eca1d","Type":"ContainerStarted","Data":"4b69b252b0b9830cbcbe6f6c9e409311eec30b2644029a42237ba45d14abe311"}
Jul 28 17:31:15.369904 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:15.369862    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd" podStartSLOduration=1.36984768 podStartE2EDuration="1.36984768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:31:15.369415387 +0000 UTC m=+1263.598614485" watchObservedRunningTime="2026-07-28 17:31:15.36984768 +0000 UTC m=+1263.599046775"
Jul 28 17:31:16.071532 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:16.071496    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:25.071658 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:25.071622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:25.084445 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:25.084422    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:25.415899 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:25.415812    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:56.121679 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.121638    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"]
Jul 28 17:31:56.131975 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:31:56.131930    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:31:56.131975 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:31:56.131975 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:31:56.131975 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59" cmd=["/bin/sleep","15"]
Jul 28 17:31:56.132340 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.131986    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd" message="" err=<
Jul 28 17:31:56.132340 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:31:56.132340 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:31:56.132340 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:31:56.132340 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:31:56.131999    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:31:56.132340 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:31:56.132340 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:31:56.132340 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd" podUID="7764fe92-58fa-4a24-874a-9ec3419eca1d" containerName="main" containerID="cri-o://6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59"
Jul 28 17:31:56.132340 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.132049    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd" podUID="7764fe92-58fa-4a24-874a-9ec3419eca1d" containerName="main" containerID="cri-o://6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59" gracePeriod=60
Jul 28 17:31:56.485232 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.485204    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:56.562595 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.562560    2579 generic.go:356] "Generic (PLEG): container finished" podID="7764fe92-58fa-4a24-874a-9ec3419eca1d" containerID="6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59" exitCode=0
Jul 28 17:31:56.562777 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.562612    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"
Jul 28 17:31:56.562777 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.562632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd" event={"ID":"7764fe92-58fa-4a24-874a-9ec3419eca1d","Type":"ContainerDied","Data":"6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59"}
Jul 28 17:31:56.562777 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.562749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd" event={"ID":"7764fe92-58fa-4a24-874a-9ec3419eca1d","Type":"ContainerDied","Data":"4b69b252b0b9830cbcbe6f6c9e409311eec30b2644029a42237ba45d14abe311"}
Jul 28 17:31:56.562884 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.562790    2579 scope.go:117] "RemoveContainer" containerID="6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59"
Jul 28 17:31:56.568526 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.568507    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-tmp-dir\") pod \"7764fe92-58fa-4a24-874a-9ec3419eca1d\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") "
Jul 28 17:31:56.568652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.568541    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-dshm\") pod \"7764fe92-58fa-4a24-874a-9ec3419eca1d\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") "
Jul 28 17:31:56.568652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.568576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-home\") pod \"7764fe92-58fa-4a24-874a-9ec3419eca1d\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") "
Jul 28 17:31:56.568652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.568615    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-model-cache\") pod \"7764fe92-58fa-4a24-874a-9ec3419eca1d\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") "
Jul 28 17:31:56.568809 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.568665    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7764fe92-58fa-4a24-874a-9ec3419eca1d-tls-certs\") pod \"7764fe92-58fa-4a24-874a-9ec3419eca1d\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") "
Jul 28 17:31:56.568809 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.568723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nsxf\" (UniqueName: \"kubernetes.io/projected/7764fe92-58fa-4a24-874a-9ec3419eca1d-kube-api-access-8nsxf\") pod \"7764fe92-58fa-4a24-874a-9ec3419eca1d\" (UID: \"7764fe92-58fa-4a24-874a-9ec3419eca1d\") "
Jul 28 17:31:56.568809 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.568789    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7764fe92-58fa-4a24-874a-9ec3419eca1d" (UID: "7764fe92-58fa-4a24-874a-9ec3419eca1d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:56.568966 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.568895    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-home" (OuterVolumeSpecName: "home") pod "7764fe92-58fa-4a24-874a-9ec3419eca1d" (UID: "7764fe92-58fa-4a24-874a-9ec3419eca1d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:56.568966 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.568904    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7764fe92-58fa-4a24-874a-9ec3419eca1d" (UID: "7764fe92-58fa-4a24-874a-9ec3419eca1d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:56.569109 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.569091    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:31:56.569169 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.569114    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:31:56.569169 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.569127    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:31:56.570768 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.570745    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-dshm" (OuterVolumeSpecName: "dshm") pod "7764fe92-58fa-4a24-874a-9ec3419eca1d" (UID: "7764fe92-58fa-4a24-874a-9ec3419eca1d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:31:56.571243 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.571202    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7764fe92-58fa-4a24-874a-9ec3419eca1d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7764fe92-58fa-4a24-874a-9ec3419eca1d" (UID: "7764fe92-58fa-4a24-874a-9ec3419eca1d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:31:56.571345 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.571328    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7764fe92-58fa-4a24-874a-9ec3419eca1d-kube-api-access-8nsxf" (OuterVolumeSpecName: "kube-api-access-8nsxf") pod "7764fe92-58fa-4a24-874a-9ec3419eca1d" (UID: "7764fe92-58fa-4a24-874a-9ec3419eca1d"). InnerVolumeSpecName "kube-api-access-8nsxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:31:56.572408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.572386    2579 scope.go:117] "RemoveContainer" containerID="6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59"
Jul 28 17:31:56.572703 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:31:56.572686    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59\": container with ID starting with 6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59 not found: ID does not exist" containerID="6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59"
Jul 28 17:31:56.572782 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.572712    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59"} err="failed to get container status \"6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59\": rpc error: code = NotFound desc = could not find container \"6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59\": container with ID starting with 6f2d0b24b8ea25fdd2c804faa18df05f1c5d51fea84b4131282cfe296fe74f59 not found: ID does not exist"
Jul 28 17:31:56.669826 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.669735    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7764fe92-58fa-4a24-874a-9ec3419eca1d-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:31:56.669826 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.669770    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nsxf\" (UniqueName: \"kubernetes.io/projected/7764fe92-58fa-4a24-874a-9ec3419eca1d-kube-api-access-8nsxf\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:31:56.669826 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.669780    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7764fe92-58fa-4a24-874a-9ec3419eca1d-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:31:56.904154 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.904119    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"]
Jul 28 17:31:56.907784 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:56.907757    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd"]
Jul 28 17:31:57.405911 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:57.405871    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-j5dwd" podUID="7764fe92-58fa-4a24-874a-9ec3419eca1d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:31:58.319700 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:31:58.319665    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7764fe92-58fa-4a24-874a-9ec3419eca1d" path="/var/lib/kubelet/pods/7764fe92-58fa-4a24-874a-9ec3419eca1d/volumes"
Jul 28 17:32:18.046409 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.046375    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"]
Jul 28 17:32:18.046936 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.046919    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7764fe92-58fa-4a24-874a-9ec3419eca1d" containerName="main"
Jul 28 17:32:18.047019 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.046937    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7764fe92-58fa-4a24-874a-9ec3419eca1d" containerName="main"
Jul 28 17:32:18.047102 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.047046    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7764fe92-58fa-4a24-874a-9ec3419eca1d" containerName="main"
Jul 28 17:32:18.051539 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.051519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.056050 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.056027    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 28 17:32:18.056186 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.056027    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 28 17:32:18.056186 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.056029    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-5g8rv\""
Jul 28 17:32:18.056186 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.056105    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 28 17:32:18.061391 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.061371    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"]
Jul 28 17:32:18.155117 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.155068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.155272 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.155167    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk598\" (UniqueName: \"kubernetes.io/projected/834955c2-a2e1-4247-8f83-6d7b247496e2-kube-api-access-xk598\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.155272 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.155204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834955c2-a2e1-4247-8f83-6d7b247496e2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.155272 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.155251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.155390 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.155308    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.155390 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.155339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.256699 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.256670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.256867 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.256712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.256867 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.256731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.256867 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.256770    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.256867 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.256836    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk598\" (UniqueName: \"kubernetes.io/projected/834955c2-a2e1-4247-8f83-6d7b247496e2-kube-api-access-xk598\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.256867 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.256863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834955c2-a2e1-4247-8f83-6d7b247496e2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.257239 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.257213    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.257404 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.257367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.257495 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.257405    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.259617 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.259597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.259784 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.259767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834955c2-a2e1-4247-8f83-6d7b247496e2-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.272742 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.272720    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk598\" (UniqueName: \"kubernetes.io/projected/834955c2-a2e1-4247-8f83-6d7b247496e2-kube-api-access-xk598\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.362815 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.362788    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:18.498611 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.498576    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"]
Jul 28 17:32:18.501527 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:32:18.501502    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834955c2_a2e1_4247_8f83_6d7b247496e2.slice/crio-e66598ad46ca4734974d377a165052a78c2d480f111f4f9e2bae75226a26dd45 WatchSource:0}: Error finding container e66598ad46ca4734974d377a165052a78c2d480f111f4f9e2bae75226a26dd45: Status 404 returned error can't find the container with id e66598ad46ca4734974d377a165052a78c2d480f111f4f9e2bae75226a26dd45
Jul 28 17:32:18.671968 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.671888    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25" event={"ID":"834955c2-a2e1-4247-8f83-6d7b247496e2","Type":"ContainerStarted","Data":"a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61"}
Jul 28 17:32:18.671968 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.671942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25" event={"ID":"834955c2-a2e1-4247-8f83-6d7b247496e2","Type":"ContainerStarted","Data":"e66598ad46ca4734974d377a165052a78c2d480f111f4f9e2bae75226a26dd45"}
Jul 28 17:32:18.707258 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:18.707204    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25" podStartSLOduration=0.707190148 podStartE2EDuration="707.190148ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:18.705141885 +0000 UTC m=+1326.934340982" watchObservedRunningTime="2026-07-28 17:32:18.707190148 +0000 UTC m=+1326.936389243"
Jul 28 17:32:19.363575 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:19.363539    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:28.363623 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:28.363580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:28.375589 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:28.375564    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:28.734789 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:28.734754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:51.713746 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:51.713671    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"]
Jul 28 17:32:51.723167 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:32:51.723135    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:32:51.723167 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:32:51.723167 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:32:51.723167 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61" cmd=["/bin/sleep","15"]
Jul 28 17:32:51.723457 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:51.723190    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25" message="" err=<
Jul 28 17:32:51.723457 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:32:51.723457 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:32:51.723457 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:32:51.723457 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:32:51.723204    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:32:51.723457 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:32:51.723457 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:32:51.723457 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25" podUID="834955c2-a2e1-4247-8f83-6d7b247496e2" containerName="main" containerID="cri-o://a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61"
Jul 28 17:32:51.723457 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:51.723265    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25" podUID="834955c2-a2e1-4247-8f83-6d7b247496e2" containerName="main" containerID="cri-o://a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61" gracePeriod=60
Jul 28 17:32:52.078646 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.078621    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:52.141850 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.141816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-dshm\") pod \"834955c2-a2e1-4247-8f83-6d7b247496e2\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") "
Jul 28 17:32:52.141850 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.141851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-home\") pod \"834955c2-a2e1-4247-8f83-6d7b247496e2\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") "
Jul 28 17:32:52.142106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.141897    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-model-cache\") pod \"834955c2-a2e1-4247-8f83-6d7b247496e2\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") "
Jul 28 17:32:52.142106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.141949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk598\" (UniqueName: \"kubernetes.io/projected/834955c2-a2e1-4247-8f83-6d7b247496e2-kube-api-access-xk598\") pod \"834955c2-a2e1-4247-8f83-6d7b247496e2\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") "
Jul 28 17:32:52.142106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.141992    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834955c2-a2e1-4247-8f83-6d7b247496e2-tls-certs\") pod \"834955c2-a2e1-4247-8f83-6d7b247496e2\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") "
Jul 28 17:32:52.142106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.142020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-tmp-dir\") pod \"834955c2-a2e1-4247-8f83-6d7b247496e2\" (UID: \"834955c2-a2e1-4247-8f83-6d7b247496e2\") "
Jul 28 17:32:52.142302 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.142222    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-home" (OuterVolumeSpecName: "home") pod "834955c2-a2e1-4247-8f83-6d7b247496e2" (UID: "834955c2-a2e1-4247-8f83-6d7b247496e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:52.142302 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.142228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "834955c2-a2e1-4247-8f83-6d7b247496e2" (UID: "834955c2-a2e1-4247-8f83-6d7b247496e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:52.142407 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.142329    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:32:52.142407 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.142343    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:32:52.142479 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.142425    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "834955c2-a2e1-4247-8f83-6d7b247496e2" (UID: "834955c2-a2e1-4247-8f83-6d7b247496e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:52.144149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.144127    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/834955c2-a2e1-4247-8f83-6d7b247496e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "834955c2-a2e1-4247-8f83-6d7b247496e2" (UID: "834955c2-a2e1-4247-8f83-6d7b247496e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:52.144248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.144191    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834955c2-a2e1-4247-8f83-6d7b247496e2-kube-api-access-xk598" (OuterVolumeSpecName: "kube-api-access-xk598") pod "834955c2-a2e1-4247-8f83-6d7b247496e2" (UID: "834955c2-a2e1-4247-8f83-6d7b247496e2"). InnerVolumeSpecName "kube-api-access-xk598". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:52.144248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.144214    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-dshm" (OuterVolumeSpecName: "dshm") pod "834955c2-a2e1-4247-8f83-6d7b247496e2" (UID: "834955c2-a2e1-4247-8f83-6d7b247496e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:52.243282 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.243183    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/834955c2-a2e1-4247-8f83-6d7b247496e2-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:32:52.243282 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.243222    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:32:52.243282 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.243231    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/834955c2-a2e1-4247-8f83-6d7b247496e2-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:32:52.243282 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.243239    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk598\" (UniqueName: \"kubernetes.io/projected/834955c2-a2e1-4247-8f83-6d7b247496e2-kube-api-access-xk598\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:32:52.835408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.835370    2579 generic.go:356] "Generic (PLEG): container finished" podID="834955c2-a2e1-4247-8f83-6d7b247496e2" containerID="a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61" exitCode=0
Jul 28 17:32:52.835866 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.835418    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"
Jul 28 17:32:52.835866 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.835459    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25" event={"ID":"834955c2-a2e1-4247-8f83-6d7b247496e2","Type":"ContainerDied","Data":"a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61"}
Jul 28 17:32:52.835866 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.835516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25" event={"ID":"834955c2-a2e1-4247-8f83-6d7b247496e2","Type":"ContainerDied","Data":"e66598ad46ca4734974d377a165052a78c2d480f111f4f9e2bae75226a26dd45"}
Jul 28 17:32:52.835866 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.835541    2579 scope.go:117] "RemoveContainer" containerID="a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61"
Jul 28 17:32:52.844051 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.844009    2579 scope.go:117] "RemoveContainer" containerID="a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61"
Jul 28 17:32:52.844283 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:32:52.844264    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61\": container with ID starting with a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61 not found: ID does not exist" containerID="a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61"
Jul 28 17:32:52.844362 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.844295    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61"} err="failed to get container status \"a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61\": rpc error: code = NotFound desc = could not find container \"a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61\": container with ID starting with a61af61bbd71d844629f97f65d8f33c6a05d646a28b0c0c32c229694825e9a61 not found: ID does not exist"
Jul 28 17:32:52.853000 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.852979    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"]
Jul 28 17:32:52.859139 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:52.859117    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94ftvw25"]
Jul 28 17:32:54.319810 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:32:54.319775    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834955c2-a2e1-4247-8f83-6d7b247496e2" path="/var/lib/kubelet/pods/834955c2-a2e1-4247-8f83-6d7b247496e2/volumes"
Jul 28 17:33:21.040787 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.040748    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"]
Jul 28 17:33:21.041385 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.041364    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834955c2-a2e1-4247-8f83-6d7b247496e2" containerName="main"
Jul 28 17:33:21.041385 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.041389    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="834955c2-a2e1-4247-8f83-6d7b247496e2" containerName="main"
Jul 28 17:33:21.041488 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.041476    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="834955c2-a2e1-4247-8f83-6d7b247496e2" containerName="main"
Jul 28 17:33:21.045167 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.045147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.047807 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.047782    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 28 17:33:21.049105 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.049068    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-hdwrw\""
Jul 28 17:33:21.049210 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.049068    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 28 17:33:21.049210 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.049069    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 28 17:33:21.054368 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.054346    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"]
Jul 28 17:33:21.203297 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.203249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.203494 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.203323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l54n9\" (UniqueName: \"kubernetes.io/projected/5fe47087-6d7f-4164-b756-3e78f014e813-kube-api-access-l54n9\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.203494 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.203379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe47087-6d7f-4164-b756-3e78f014e813-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.203494 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.203464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.203626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.203503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.203626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.203522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.304211 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.304128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe47087-6d7f-4164-b756-3e78f014e813-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.304211 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.304184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.304428 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.304362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.304428 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.304402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.304535 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.304450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.304535 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.304496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l54n9\" (UniqueName: \"kubernetes.io/projected/5fe47087-6d7f-4164-b756-3e78f014e813-kube-api-access-l54n9\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.304631 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.304573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.304948 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.304923    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.305124 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.305106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.306693 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.306671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.307391 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.307368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe47087-6d7f-4164-b756-3e78f014e813-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.316466 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.316440    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l54n9\" (UniqueName: \"kubernetes.io/projected/5fe47087-6d7f-4164-b756-3e78f014e813-kube-api-access-l54n9\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.358098 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.358048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:21.497884 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.497860    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"]
Jul 28 17:33:21.499927 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:33:21.499896    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe47087_6d7f_4164_b756_3e78f014e813.slice/crio-f63e5f8bd01261ded13b66bd72c34157f42e582fdde34b5b30a221742ea03ee9 WatchSource:0}: Error finding container f63e5f8bd01261ded13b66bd72c34157f42e582fdde34b5b30a221742ea03ee9: Status 404 returned error can't find the container with id f63e5f8bd01261ded13b66bd72c34157f42e582fdde34b5b30a221742ea03ee9
Jul 28 17:33:21.971718 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.971666    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v" event={"ID":"5fe47087-6d7f-4164-b756-3e78f014e813","Type":"ContainerStarted","Data":"6aadd9fb795caf63eb95080fde5871711cf612a321f1f28547cabb057f4b4514"}
Jul 28 17:33:21.971928 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.971740    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v" event={"ID":"5fe47087-6d7f-4164-b756-3e78f014e813","Type":"ContainerStarted","Data":"f63e5f8bd01261ded13b66bd72c34157f42e582fdde34b5b30a221742ea03ee9"}
Jul 28 17:33:21.991566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:21.991525    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v" podStartSLOduration=0.991509656 podStartE2EDuration="991.509656ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:33:21.989624812 +0000 UTC m=+1390.218823909" watchObservedRunningTime="2026-07-28 17:33:21.991509656 +0000 UTC m=+1390.220708785"
Jul 28 17:33:22.358178 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:22.358149    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:31.358929 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:31.358890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:31.371174 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:31.371152    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:32.029678 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:32.029647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:54.903853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:54.903823    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"]
Jul 28 17:33:54.913232 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:33:54.913197    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:33:54.913232 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:33:54.913232 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:33:54.913232 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="6aadd9fb795caf63eb95080fde5871711cf612a321f1f28547cabb057f4b4514" cmd=["/bin/sleep","15"]
Jul 28 17:33:54.913481 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:54.913244    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v" message="" err=<
Jul 28 17:33:54.913481 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:33:54.913481 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:33:54.913481 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:33:54.913481 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:33:54.913257    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:33:54.913481 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:33:54.913481 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:33:54.913481 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v" podUID="5fe47087-6d7f-4164-b756-3e78f014e813" containerName="main" containerID="cri-o://6aadd9fb795caf63eb95080fde5871711cf612a321f1f28547cabb057f4b4514"
Jul 28 17:33:54.913481 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:54.913289    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v" podUID="5fe47087-6d7f-4164-b756-3e78f014e813" containerName="main" containerID="cri-o://6aadd9fb795caf63eb95080fde5871711cf612a321f1f28547cabb057f4b4514" gracePeriod=60
Jul 28 17:33:55.018629 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.018591    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v" podUID="5fe47087-6d7f-4164-b756-3e78f014e813" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/health\": dial tcp 10.132.0.40:8000: connect: connection refused"
Jul 28 17:33:55.132037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.132002    2579 generic.go:356] "Generic (PLEG): container finished" podID="5fe47087-6d7f-4164-b756-3e78f014e813" containerID="6aadd9fb795caf63eb95080fde5871711cf612a321f1f28547cabb057f4b4514" exitCode=0
Jul 28 17:33:55.132176 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.132057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v" event={"ID":"5fe47087-6d7f-4164-b756-3e78f014e813","Type":"ContainerDied","Data":"6aadd9fb795caf63eb95080fde5871711cf612a321f1f28547cabb057f4b4514"}
Jul 28 17:33:55.261588 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.261562    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:55.413984 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.413949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-home\") pod \"5fe47087-6d7f-4164-b756-3e78f014e813\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") "
Jul 28 17:33:55.413984 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.413984    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-dshm\") pod \"5fe47087-6d7f-4164-b756-3e78f014e813\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") "
Jul 28 17:33:55.414271 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.414011    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l54n9\" (UniqueName: \"kubernetes.io/projected/5fe47087-6d7f-4164-b756-3e78f014e813-kube-api-access-l54n9\") pod \"5fe47087-6d7f-4164-b756-3e78f014e813\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") "
Jul 28 17:33:55.414271 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.414047    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe47087-6d7f-4164-b756-3e78f014e813-tls-certs\") pod \"5fe47087-6d7f-4164-b756-3e78f014e813\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") "
Jul 28 17:33:55.414271 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.414065    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-tmp-dir\") pod \"5fe47087-6d7f-4164-b756-3e78f014e813\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") "
Jul 28 17:33:55.414271 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.414123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-model-cache\") pod \"5fe47087-6d7f-4164-b756-3e78f014e813\" (UID: \"5fe47087-6d7f-4164-b756-3e78f014e813\") "
Jul 28 17:33:55.414475 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.414314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-home" (OuterVolumeSpecName: "home") pod "5fe47087-6d7f-4164-b756-3e78f014e813" (UID: "5fe47087-6d7f-4164-b756-3e78f014e813"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:33:55.414475 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.414371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5fe47087-6d7f-4164-b756-3e78f014e813" (UID: "5fe47087-6d7f-4164-b756-3e78f014e813"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:33:55.414475 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.414396    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-model-cache" (OuterVolumeSpecName: "model-cache") pod "5fe47087-6d7f-4164-b756-3e78f014e813" (UID: "5fe47087-6d7f-4164-b756-3e78f014e813"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:33:55.416261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.416232    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-dshm" (OuterVolumeSpecName: "dshm") pod "5fe47087-6d7f-4164-b756-3e78f014e813" (UID: "5fe47087-6d7f-4164-b756-3e78f014e813"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:33:55.416372 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.416313    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe47087-6d7f-4164-b756-3e78f014e813-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5fe47087-6d7f-4164-b756-3e78f014e813" (UID: "5fe47087-6d7f-4164-b756-3e78f014e813"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:33:55.416372 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.416327    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe47087-6d7f-4164-b756-3e78f014e813-kube-api-access-l54n9" (OuterVolumeSpecName: "kube-api-access-l54n9") pod "5fe47087-6d7f-4164-b756-3e78f014e813" (UID: "5fe47087-6d7f-4164-b756-3e78f014e813"). InnerVolumeSpecName "kube-api-access-l54n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:33:55.515262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.515169    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:33:55.515262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.515196    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:33:55.515262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.515208    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l54n9\" (UniqueName: \"kubernetes.io/projected/5fe47087-6d7f-4164-b756-3e78f014e813-kube-api-access-l54n9\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:33:55.515262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.515220    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fe47087-6d7f-4164-b756-3e78f014e813-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:33:55.515262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.515233    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:33:55.515262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:55.515244    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fe47087-6d7f-4164-b756-3e78f014e813-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:33:56.138177 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:56.138146    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"
Jul 28 17:33:56.138177 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:56.138156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v" event={"ID":"5fe47087-6d7f-4164-b756-3e78f014e813","Type":"ContainerDied","Data":"f63e5f8bd01261ded13b66bd72c34157f42e582fdde34b5b30a221742ea03ee9"}
Jul 28 17:33:56.138795 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:56.138217    2579 scope.go:117] "RemoveContainer" containerID="6aadd9fb795caf63eb95080fde5871711cf612a321f1f28547cabb057f4b4514"
Jul 28 17:33:56.162627 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:56.162601    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"]
Jul 28 17:33:56.166351 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:56.166327    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-6tc9v"]
Jul 28 17:33:56.320019 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:33:56.319986    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fe47087-6d7f-4164-b756-3e78f014e813" path="/var/lib/kubelet/pods/5fe47087-6d7f-4164-b756-3e78f014e813/volumes"
Jul 28 17:34:22.247447 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.247417    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"]
Jul 28 17:34:22.247884 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.247783    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fe47087-6d7f-4164-b756-3e78f014e813" containerName="main"
Jul 28 17:34:22.247884 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.247794    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fe47087-6d7f-4164-b756-3e78f014e813" containerName="main"
Jul 28 17:34:22.247884 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.247861    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fe47087-6d7f-4164-b756-3e78f014e813" containerName="main"
Jul 28 17:34:22.252244 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.252224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.256891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.256870    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-g922c\""
Jul 28 17:34:22.257029 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.256870    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 28 17:34:22.257029 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.256904    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 28 17:34:22.257029 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.256906    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 28 17:34:22.262730 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.262711    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"]
Jul 28 17:34:22.447969 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.447930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87476e3-4c56-48fc-b974-177e1113399b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.448195 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.448027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.448195 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.448135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.448195 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.448171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.448334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.448202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxsp\" (UniqueName: \"kubernetes.io/projected/a87476e3-4c56-48fc-b974-177e1113399b-kube-api-access-bdxsp\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.448334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.448223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.512039 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.511961    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"]
Jul 28 17:34:22.515359 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.515343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:34:22.517719 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.517693    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-9wb52\""
Jul 28 17:34:22.528502 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.528477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"]
Jul 28 17:34:22.548837 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.548812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74449a1a-1f16-4d94-aad0-378b345e91d5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5\" (UID: \"74449a1a-1f16-4d94-aad0-378b345e91d5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:34:22.548967 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.548844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87476e3-4c56-48fc-b974-177e1113399b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.548967 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.548888    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.548967 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.548918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdxj\" (UniqueName: \"kubernetes.io/projected/74449a1a-1f16-4d94-aad0-378b345e91d5-kube-api-access-5mdxj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5\" (UID: \"74449a1a-1f16-4d94-aad0-378b345e91d5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:34:22.548967 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.548936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.548967 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.548956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.549265 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.548976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxsp\" (UniqueName: \"kubernetes.io/projected/a87476e3-4c56-48fc-b974-177e1113399b-kube-api-access-bdxsp\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.549265 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.549001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.549377 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.549343    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.549438 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.549414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.549478 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.549437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.551142 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.551126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.551238 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.551224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87476e3-4c56-48fc-b974-177e1113399b-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.561612 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.561587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxsp\" (UniqueName: \"kubernetes.io/projected/a87476e3-4c56-48fc-b974-177e1113399b-kube-api-access-bdxsp\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-whnk2\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.562926 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.562906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:22.649913 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.649883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdxj\" (UniqueName: \"kubernetes.io/projected/74449a1a-1f16-4d94-aad0-378b345e91d5-kube-api-access-5mdxj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5\" (UID: \"74449a1a-1f16-4d94-aad0-378b345e91d5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:34:22.650063 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.649938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74449a1a-1f16-4d94-aad0-378b345e91d5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5\" (UID: \"74449a1a-1f16-4d94-aad0-378b345e91d5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:34:22.652380 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.652345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74449a1a-1f16-4d94-aad0-378b345e91d5-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5\" (UID: \"74449a1a-1f16-4d94-aad0-378b345e91d5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:34:22.659149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.659119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdxj\" (UniqueName: \"kubernetes.io/projected/74449a1a-1f16-4d94-aad0-378b345e91d5-kube-api-access-5mdxj\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5\" (UID: \"74449a1a-1f16-4d94-aad0-378b345e91d5\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:34:22.703830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.703808    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"]
Jul 28 17:34:22.705745 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:34:22.705721    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda87476e3_4c56_48fc_b974_177e1113399b.slice/crio-f80fcd2ef0f519ceebf405afddcc45d9d48d2ae0a1f4e0bbf54e86b7190ef703 WatchSource:0}: Error finding container f80fcd2ef0f519ceebf405afddcc45d9d48d2ae0a1f4e0bbf54e86b7190ef703: Status 404 returned error can't find the container with id f80fcd2ef0f519ceebf405afddcc45d9d48d2ae0a1f4e0bbf54e86b7190ef703
Jul 28 17:34:22.825897 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.825865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:34:22.959139 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.959112    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"]
Jul 28 17:34:22.961474 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:34:22.961447    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74449a1a_1f16_4d94_aad0_378b345e91d5.slice/crio-19757c042eb1794b039c6f33898eb3a700f5d8f28e3ad05b0a9741accf68f4cc WatchSource:0}: Error finding container 19757c042eb1794b039c6f33898eb3a700f5d8f28e3ad05b0a9741accf68f4cc: Status 404 returned error can't find the container with id 19757c042eb1794b039c6f33898eb3a700f5d8f28e3ad05b0a9741accf68f4cc
Jul 28 17:34:22.963413 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:22.963391    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:34:23.268721 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:23.268625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5" event={"ID":"74449a1a-1f16-4d94-aad0-378b345e91d5","Type":"ContainerStarted","Data":"19757c042eb1794b039c6f33898eb3a700f5d8f28e3ad05b0a9741accf68f4cc"}
Jul 28 17:34:23.270234 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:23.270205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2" event={"ID":"a87476e3-4c56-48fc-b974-177e1113399b","Type":"ContainerStarted","Data":"7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e"}
Jul 28 17:34:23.270359 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:23.270251    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2" event={"ID":"a87476e3-4c56-48fc-b974-177e1113399b","Type":"ContainerStarted","Data":"f80fcd2ef0f519ceebf405afddcc45d9d48d2ae0a1f4e0bbf54e86b7190ef703"}
Jul 28 17:34:23.288347 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:23.288303    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2" podStartSLOduration=1.288286714 podStartE2EDuration="1.288286714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:23.287521947 +0000 UTC m=+1451.516721043" watchObservedRunningTime="2026-07-28 17:34:23.288286714 +0000 UTC m=+1451.517485810"
Jul 28 17:34:23.563378 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:23.563274    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:25.281774 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:25.281731    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5" event={"ID":"74449a1a-1f16-4d94-aad0-378b345e91d5","Type":"ContainerStarted","Data":"593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf"}
Jul 28 17:34:25.282194 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:25.282024    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:34:25.300903 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:25.300861    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5" podStartSLOduration=1.140875642 podStartE2EDuration="3.300847139s" totalImagesPullingTime="2.159971497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:25.299591482 +0000 UTC m=+1453.528790579" watchObservedRunningTime="2026-07-28 17:34:25.300847139 +0000 UTC m=+1453.530046235"
Jul 28 17:34:32.563575 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:32.563529    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:32.576312 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:32.576287    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:33.330863 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:33.330835    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:55.774451 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:55.774419    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"]
Jul 28 17:34:55.782791 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:55.782760    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"]
Jul 28 17:34:55.784462 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:55.784429    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5" podUID="74449a1a-1f16-4d94-aad0-378b345e91d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:34:55.784516    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e" cmd=["/bin/sleep","15"]
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:55.784567    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2" message="" err=<
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:34:55.784585    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:34:55.784599 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2" podUID="a87476e3-4c56-48fc-b974-177e1113399b" containerName="main" containerID="cri-o://7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e"
Jul 28 17:34:55.785043 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:55.784624    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2" podUID="a87476e3-4c56-48fc-b974-177e1113399b" containerName="main" containerID="cri-o://7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e" gracePeriod=60
Jul 28 17:34:56.136526 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.136500    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:56.263282 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.263242    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87476e3-4c56-48fc-b974-177e1113399b-tls-certs\") pod \"a87476e3-4c56-48fc-b974-177e1113399b\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") "
Jul 28 17:34:56.263446 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.263292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdxsp\" (UniqueName: \"kubernetes.io/projected/a87476e3-4c56-48fc-b974-177e1113399b-kube-api-access-bdxsp\") pod \"a87476e3-4c56-48fc-b974-177e1113399b\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") "
Jul 28 17:34:56.263446 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.263376    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-home\") pod \"a87476e3-4c56-48fc-b974-177e1113399b\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") "
Jul 28 17:34:56.263446 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.263394    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-tmp-dir\") pod \"a87476e3-4c56-48fc-b974-177e1113399b\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") "
Jul 28 17:34:56.263446 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.263415    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-dshm\") pod \"a87476e3-4c56-48fc-b974-177e1113399b\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") "
Jul 28 17:34:56.263446 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.263432    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-model-cache\") pod \"a87476e3-4c56-48fc-b974-177e1113399b\" (UID: \"a87476e3-4c56-48fc-b974-177e1113399b\") "
Jul 28 17:34:56.263720 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.263607    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-home" (OuterVolumeSpecName: "home") pod "a87476e3-4c56-48fc-b974-177e1113399b" (UID: "a87476e3-4c56-48fc-b974-177e1113399b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:34:56.263779 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.263699    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a87476e3-4c56-48fc-b974-177e1113399b" (UID: "a87476e3-4c56-48fc-b974-177e1113399b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:34:56.263779 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.263756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a87476e3-4c56-48fc-b974-177e1113399b" (UID: "a87476e3-4c56-48fc-b974-177e1113399b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:34:56.265494 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.265471    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-dshm" (OuterVolumeSpecName: "dshm") pod "a87476e3-4c56-48fc-b974-177e1113399b" (UID: "a87476e3-4c56-48fc-b974-177e1113399b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:34:56.265582 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.265536    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a87476e3-4c56-48fc-b974-177e1113399b-kube-api-access-bdxsp" (OuterVolumeSpecName: "kube-api-access-bdxsp") pod "a87476e3-4c56-48fc-b974-177e1113399b" (UID: "a87476e3-4c56-48fc-b974-177e1113399b"). InnerVolumeSpecName "kube-api-access-bdxsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:34:56.265625 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.265595    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a87476e3-4c56-48fc-b974-177e1113399b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a87476e3-4c56-48fc-b974-177e1113399b" (UID: "a87476e3-4c56-48fc-b974-177e1113399b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:34:56.364176 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.364146    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a87476e3-4c56-48fc-b974-177e1113399b-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.364176 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.364170    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdxsp\" (UniqueName: \"kubernetes.io/projected/a87476e3-4c56-48fc-b974-177e1113399b-kube-api-access-bdxsp\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.364332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.364181    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.364332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.364206    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.364332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.364215    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.364332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.364223    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a87476e3-4c56-48fc-b974-177e1113399b-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:34:56.431172 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.431142    2579 generic.go:356] "Generic (PLEG): container finished" podID="a87476e3-4c56-48fc-b974-177e1113399b" containerID="7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e" exitCode=0
Jul 28 17:34:56.431296 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.431177    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"
Jul 28 17:34:56.431296 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.431230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2" event={"ID":"a87476e3-4c56-48fc-b974-177e1113399b","Type":"ContainerDied","Data":"7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e"}
Jul 28 17:34:56.431296 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.431290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2" event={"ID":"a87476e3-4c56-48fc-b974-177e1113399b","Type":"ContainerDied","Data":"f80fcd2ef0f519ceebf405afddcc45d9d48d2ae0a1f4e0bbf54e86b7190ef703"}
Jul 28 17:34:56.431405 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.431314    2579 scope.go:117] "RemoveContainer" containerID="7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e"
Jul 28 17:34:56.440190 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.440176    2579 scope.go:117] "RemoveContainer" containerID="7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e"
Jul 28 17:34:56.440425 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:34:56.440406    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e\": container with ID starting with 7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e not found: ID does not exist" containerID="7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e"
Jul 28 17:34:56.440468 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.440434    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e"} err="failed to get container status \"7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e\": rpc error: code = NotFound desc = could not find container \"7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e\": container with ID starting with 7fb5fb2557f38c57b02331eb2fc44324b43aab765a64b3a5e68b0e68790af32e not found: ID does not exist"
Jul 28 17:34:56.447944 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.447898    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"]
Jul 28 17:34:56.450029 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:56.450011    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-whnk2"]
Jul 28 17:34:58.320603 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:34:58.320569    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a87476e3-4c56-48fc-b974-177e1113399b" path="/var/lib/kubelet/pods/a87476e3-4c56-48fc-b974-177e1113399b/volumes"
Jul 28 17:35:05.785073 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:05.785029    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5" podUID="74449a1a-1f16-4d94-aad0-378b345e91d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:35:10.802140 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:10.802059    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5" podUID="74449a1a-1f16-4d94-aad0-378b345e91d5" containerName="main" containerID="cri-o://593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf" gracePeriod=45
Jul 28 17:35:11.055953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.055898    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:35:11.101063 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.101032    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mdxj\" (UniqueName: \"kubernetes.io/projected/74449a1a-1f16-4d94-aad0-378b345e91d5-kube-api-access-5mdxj\") pod \"74449a1a-1f16-4d94-aad0-378b345e91d5\" (UID: \"74449a1a-1f16-4d94-aad0-378b345e91d5\") "
Jul 28 17:35:11.101236 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.101103    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74449a1a-1f16-4d94-aad0-378b345e91d5-tls-certs\") pod \"74449a1a-1f16-4d94-aad0-378b345e91d5\" (UID: \"74449a1a-1f16-4d94-aad0-378b345e91d5\") "
Jul 28 17:35:11.103127 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.103101    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74449a1a-1f16-4d94-aad0-378b345e91d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74449a1a-1f16-4d94-aad0-378b345e91d5" (UID: "74449a1a-1f16-4d94-aad0-378b345e91d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:35:11.103229 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.103177    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74449a1a-1f16-4d94-aad0-378b345e91d5-kube-api-access-5mdxj" (OuterVolumeSpecName: "kube-api-access-5mdxj") pod "74449a1a-1f16-4d94-aad0-378b345e91d5" (UID: "74449a1a-1f16-4d94-aad0-378b345e91d5"). InnerVolumeSpecName "kube-api-access-5mdxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:35:11.201868 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.201829    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mdxj\" (UniqueName: \"kubernetes.io/projected/74449a1a-1f16-4d94-aad0-378b345e91d5-kube-api-access-5mdxj\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:35:11.201868 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.201859    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74449a1a-1f16-4d94-aad0-378b345e91d5-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:35:11.503965 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.503926    2579 generic.go:356] "Generic (PLEG): container finished" podID="74449a1a-1f16-4d94-aad0-378b345e91d5" containerID="593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf" exitCode=0
Jul 28 17:35:11.504171 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.503973    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"
Jul 28 17:35:11.504171 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.503976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5" event={"ID":"74449a1a-1f16-4d94-aad0-378b345e91d5","Type":"ContainerDied","Data":"593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf"}
Jul 28 17:35:11.504171 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.504032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5" event={"ID":"74449a1a-1f16-4d94-aad0-378b345e91d5","Type":"ContainerDied","Data":"19757c042eb1794b039c6f33898eb3a700f5d8f28e3ad05b0a9741accf68f4cc"}
Jul 28 17:35:11.504171 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.504062    2579 scope.go:117] "RemoveContainer" containerID="593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf"
Jul 28 17:35:11.512863 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.512842    2579 scope.go:117] "RemoveContainer" containerID="593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf"
Jul 28 17:35:11.513145 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:35:11.513125    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf\": container with ID starting with 593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf not found: ID does not exist" containerID="593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf"
Jul 28 17:35:11.513219 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.513153    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf"} err="failed to get container status \"593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf\": rpc error: code = NotFound desc = could not find container \"593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf\": container with ID starting with 593a89279557db08b6e9241b3894503dd7cc879ddec3badfcd513d56848ff9cf not found: ID does not exist"
Jul 28 17:35:11.525460 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.525435    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"]
Jul 28 17:35:11.528914 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:11.528892    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-66cb9cfk7tw5"]
Jul 28 17:35:12.296643 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:12.296608    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:35:12.300881 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:12.300856    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:35:12.320158 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:12.320128    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74449a1a-1f16-4d94-aad0-378b345e91d5" path="/var/lib/kubelet/pods/74449a1a-1f16-4d94-aad0-378b345e91d5/volumes"
Jul 28 17:35:38.549523 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.549445    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"]
Jul 28 17:35:38.549995 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.549887    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a87476e3-4c56-48fc-b974-177e1113399b" containerName="main"
Jul 28 17:35:38.549995 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.549907    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a87476e3-4c56-48fc-b974-177e1113399b" containerName="main"
Jul 28 17:35:38.549995 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.549943    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74449a1a-1f16-4d94-aad0-378b345e91d5" containerName="main"
Jul 28 17:35:38.549995 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.549949    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="74449a1a-1f16-4d94-aad0-378b345e91d5" containerName="main"
Jul 28 17:35:38.550164 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.550012    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a87476e3-4c56-48fc-b974-177e1113399b" containerName="main"
Jul 28 17:35:38.550164 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.550024    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="74449a1a-1f16-4d94-aad0-378b345e91d5" containerName="main"
Jul 28 17:35:38.552736 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.552721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.556693 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.556667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-knhq8\""
Jul 28 17:35:38.556693 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.556669    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 28 17:35:38.556884 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.556700    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 28 17:35:38.556884 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.556667    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 28 17:35:38.564072 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.564051    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"]
Jul 28 17:35:38.659475 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.659441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.659658 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.659489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.659658 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.659603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.659753 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.659672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pvhl\" (UniqueName: \"kubernetes.io/projected/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-kube-api-access-9pvhl\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.659753 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.659699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.659753 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.659719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.761035 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.761001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pvhl\" (UniqueName: \"kubernetes.io/projected/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-kube-api-access-9pvhl\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.761268 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.761050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.761268 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.761099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.761268 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.761128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.761268 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.761165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.761268 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.761239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.761543 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.761494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.761604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.761546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.761658 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.761615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.763449 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.763429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.763672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.763655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.770271 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.770244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pvhl\" (UniqueName: \"kubernetes.io/projected/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-kube-api-access-9pvhl\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-vvng2\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.854782 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.854752    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"]
Jul 28 17:35:38.858579 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.858562    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:38.861444 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.861421    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-fbfc9\""
Jul 28 17:35:38.864041 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.864023    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:38.868382 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.868361    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"]
Jul 28 17:35:38.962776 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.962733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbd54\" (UniqueName: \"kubernetes.io/projected/687b42a4-f849-499c-9a1d-78bedb9c673c-kube-api-access-nbd54\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:38.962932 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.962793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687b42a4-f849-499c-9a1d-78bedb9c673c-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:38.962932 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.962849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:38.962932 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.962905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:38.997397 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:38.997369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"]
Jul 28 17:35:38.999479 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:35:38.999448    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcefa1bbf_2d38_4d59_b9dd_6d5517d838ad.slice/crio-9db33248ff4bdfda9142822468d47d52838551a88e65210182a4d8f6c228f69d WatchSource:0}: Error finding container 9db33248ff4bdfda9142822468d47d52838551a88e65210182a4d8f6c228f69d: Status 404 returned error can't find the container with id 9db33248ff4bdfda9142822468d47d52838551a88e65210182a4d8f6c228f69d
Jul 28 17:35:39.063562 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.063536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:39.063683 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.063627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbd54\" (UniqueName: \"kubernetes.io/projected/687b42a4-f849-499c-9a1d-78bedb9c673c-kube-api-access-nbd54\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:39.063683 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.063653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687b42a4-f849-499c-9a1d-78bedb9c673c-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:39.063683 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.063671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:39.063972 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.063953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:39.064020 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.063988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:39.065898 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.065879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687b42a4-f849-499c-9a1d-78bedb9c673c-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:39.071347 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.071325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbd54\" (UniqueName: \"kubernetes.io/projected/687b42a4-f849-499c-9a1d-78bedb9c673c-kube-api-access-nbd54\") pod \"scheduler-custom-template-test-kserve-router-scheduler-795b8ljv\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:39.169940 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.169841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:39.306715 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.306683    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"]
Jul 28 17:35:39.310032 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:35:39.310010    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687b42a4_f849_499c_9a1d_78bedb9c673c.slice/crio-6c0104951898a4b0bfc7e7bc259259bdfd3d7b02c311960ef41d6a439a965787 WatchSource:0}: Error finding container 6c0104951898a4b0bfc7e7bc259259bdfd3d7b02c311960ef41d6a439a965787: Status 404 returned error can't find the container with id 6c0104951898a4b0bfc7e7bc259259bdfd3d7b02c311960ef41d6a439a965787
Jul 28 17:35:39.633509 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.633473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv" event={"ID":"687b42a4-f849-499c-9a1d-78bedb9c673c","Type":"ContainerStarted","Data":"eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a"}
Jul 28 17:35:39.633987 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.633647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:35:39.633987 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.633704    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv" event={"ID":"687b42a4-f849-499c-9a1d-78bedb9c673c","Type":"ContainerStarted","Data":"6c0104951898a4b0bfc7e7bc259259bdfd3d7b02c311960ef41d6a439a965787"}
Jul 28 17:35:39.635015 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.634992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2" event={"ID":"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad","Type":"ContainerStarted","Data":"10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1"}
Jul 28 17:35:39.635164 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.635030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2" event={"ID":"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad","Type":"ContainerStarted","Data":"9db33248ff4bdfda9142822468d47d52838551a88e65210182a4d8f6c228f69d"}
Jul 28 17:35:39.653699 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.653646    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv" podStartSLOduration=1.6536287 podStartE2EDuration="1.6536287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:39.652507446 +0000 UTC m=+1527.881706553" watchObservedRunningTime="2026-07-28 17:35:39.6536287 +0000 UTC m=+1527.882827796"
Jul 28 17:35:39.675910 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.675861    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2" podStartSLOduration=1.675846824 podStartE2EDuration="1.675846824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:35:39.673892126 +0000 UTC m=+1527.903091230" watchObservedRunningTime="2026-07-28 17:35:39.675846824 +0000 UTC m=+1527.905045960"
Jul 28 17:35:39.865187 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:39.865142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:48.865139 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:48.865103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:48.877473 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:48.877451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:35:49.693692 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:35:49.693663    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:36:10.641446 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:10.641419    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:36:12.253305 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.253271    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"]
Jul 28 17:36:12.260194 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.260165    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"]
Jul 28 17:36:12.264408 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:36:12.264375    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:36:12.264408 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:36:12.264408 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:36:12.264408 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1" cmd=["/bin/sleep","15"]
Jul 28 17:36:12.264740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.264422    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2" message="" err=<
Jul 28 17:36:12.264740 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:36:12.264740 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:36:12.264740 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:36:12.264740 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:36:12.264435    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:36:12.264740 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:36:12.264740 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:36:12.264740 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2" podUID="cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" containerName="main" containerID="cri-o://10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1"
Jul 28 17:36:12.264740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.264460    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2" podUID="cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" containerName="main" containerID="cri-o://10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1" gracePeriod=60
Jul 28 17:36:12.625060 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.625037    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:36:12.659758 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.659733    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-model-cache\") pod \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") "
Jul 28 17:36:12.659912 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.659776    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tls-certs\") pod \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") "
Jul 28 17:36:12.659912 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.659818    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-home\") pod \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") "
Jul 28 17:36:12.659912 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.659860    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tmp-dir\") pod \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") "
Jul 28 17:36:12.659912 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.659903    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pvhl\" (UniqueName: \"kubernetes.io/projected/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-kube-api-access-9pvhl\") pod \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") "
Jul 28 17:36:12.660152 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.659996    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-dshm\") pod \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\" (UID: \"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad\") "
Jul 28 17:36:12.660152 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.660008    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" (UID: "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:36:12.660251 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.660228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-home" (OuterVolumeSpecName: "home") pod "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" (UID: "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:36:12.660332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.660314    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:36:12.660382 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.660339    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:36:12.660448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.660401    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" (UID: "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:36:12.662579 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.662549    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-dshm" (OuterVolumeSpecName: "dshm") pod "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" (UID: "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:36:12.662694 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.662611    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-kube-api-access-9pvhl" (OuterVolumeSpecName: "kube-api-access-9pvhl") pod "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" (UID: "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad"). InnerVolumeSpecName "kube-api-access-9pvhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:36:12.662694 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.662621    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" (UID: "cefa1bbf-2d38-4d59-b9dd-6d5517d838ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:36:12.761145 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.761107    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:36:12.761145 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.761136    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:36:12.761145 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.761148    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pvhl\" (UniqueName: \"kubernetes.io/projected/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-kube-api-access-9pvhl\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:36:12.761366 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.761160    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:36:12.802210 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.802132    2579 generic.go:356] "Generic (PLEG): container finished" podID="cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" containerID="10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1" exitCode=0
Jul 28 17:36:12.802210 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.802176    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"
Jul 28 17:36:12.802406 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.802215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2" event={"ID":"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad","Type":"ContainerDied","Data":"10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1"}
Jul 28 17:36:12.802406 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.802267    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2" event={"ID":"cefa1bbf-2d38-4d59-b9dd-6d5517d838ad","Type":"ContainerDied","Data":"9db33248ff4bdfda9142822468d47d52838551a88e65210182a4d8f6c228f69d"}
Jul 28 17:36:12.802406 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.802298    2579 scope.go:117] "RemoveContainer" containerID="10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1"
Jul 28 17:36:12.811324 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.811307    2579 scope.go:117] "RemoveContainer" containerID="10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1"
Jul 28 17:36:12.811559 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:36:12.811543    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1\": container with ID starting with 10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1 not found: ID does not exist" containerID="10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1"
Jul 28 17:36:12.811626 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.811566    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1"} err="failed to get container status \"10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1\": rpc error: code = NotFound desc = could not find container \"10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1\": container with ID starting with 10378ed2f778f402a6c1e01e4d243a9db172beb2671d6a8b3b87f0ecfecbe6a1 not found: ID does not exist"
Jul 28 17:36:12.826811 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.826777    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"]
Jul 28 17:36:12.828594 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:12.828575    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-vvng2"]
Jul 28 17:36:14.319830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:14.319796    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" path="/var/lib/kubelet/pods/cefa1bbf-2d38-4d59-b9dd-6d5517d838ad/volumes"
Jul 28 17:36:20.641505 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:20.641468    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv" podUID="687b42a4-f849-499c-9a1d-78bedb9c673c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:36:27.279547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.279505    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv" podUID="687b42a4-f849-499c-9a1d-78bedb9c673c" containerName="main" containerID="cri-o://eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a" gracePeriod=45
Jul 28 17:36:27.522584 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.522549    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:36:27.599306 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.599267    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-cachi2-cache\") pod \"687b42a4-f849-499c-9a1d-78bedb9c673c\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") "
Jul 28 17:36:27.599483 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.599371    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687b42a4-f849-499c-9a1d-78bedb9c673c-tls-certs\") pod \"687b42a4-f849-499c-9a1d-78bedb9c673c\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") "
Jul 28 17:36:27.599483 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.599423    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-tokenizer-cache\") pod \"687b42a4-f849-499c-9a1d-78bedb9c673c\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") "
Jul 28 17:36:27.599599 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.599502    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nbd54\" (UniqueName: \"kubernetes.io/projected/687b42a4-f849-499c-9a1d-78bedb9c673c-kube-api-access-nbd54\") pod \"687b42a4-f849-499c-9a1d-78bedb9c673c\" (UID: \"687b42a4-f849-499c-9a1d-78bedb9c673c\") "
Jul 28 17:36:27.599599 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.599508    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "687b42a4-f849-499c-9a1d-78bedb9c673c" (UID: "687b42a4-f849-499c-9a1d-78bedb9c673c"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:36:27.599716 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.599693    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "687b42a4-f849-499c-9a1d-78bedb9c673c" (UID: "687b42a4-f849-499c-9a1d-78bedb9c673c"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:36:27.599774 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.599756    2579 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-tokenizer-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:36:27.599820 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.599776    2579 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/687b42a4-f849-499c-9a1d-78bedb9c673c-cachi2-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:36:27.601442 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.601419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687b42a4-f849-499c-9a1d-78bedb9c673c-kube-api-access-nbd54" (OuterVolumeSpecName: "kube-api-access-nbd54") pod "687b42a4-f849-499c-9a1d-78bedb9c673c" (UID: "687b42a4-f849-499c-9a1d-78bedb9c673c"). InnerVolumeSpecName "kube-api-access-nbd54". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:36:27.601547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.601489    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687b42a4-f849-499c-9a1d-78bedb9c673c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "687b42a4-f849-499c-9a1d-78bedb9c673c" (UID: "687b42a4-f849-499c-9a1d-78bedb9c673c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:36:27.701216 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.701185    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbd54\" (UniqueName: \"kubernetes.io/projected/687b42a4-f849-499c-9a1d-78bedb9c673c-kube-api-access-nbd54\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:36:27.701216 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.701214    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/687b42a4-f849-499c-9a1d-78bedb9c673c-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:36:27.877134 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.877022    2579 generic.go:356] "Generic (PLEG): container finished" podID="687b42a4-f849-499c-9a1d-78bedb9c673c" containerID="eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a" exitCode=0
Jul 28 17:36:27.877134 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.877098    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"
Jul 28 17:36:27.877134 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.877110    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv" event={"ID":"687b42a4-f849-499c-9a1d-78bedb9c673c","Type":"ContainerDied","Data":"eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a"}
Jul 28 17:36:27.877367 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.877198    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv" event={"ID":"687b42a4-f849-499c-9a1d-78bedb9c673c","Type":"ContainerDied","Data":"6c0104951898a4b0bfc7e7bc259259bdfd3d7b02c311960ef41d6a439a965787"}
Jul 28 17:36:27.877367 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.877229    2579 scope.go:117] "RemoveContainer" containerID="eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a"
Jul 28 17:36:27.886857 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.886834    2579 scope.go:117] "RemoveContainer" containerID="eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a"
Jul 28 17:36:27.887174 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:36:27.887155    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a\": container with ID starting with eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a not found: ID does not exist" containerID="eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a"
Jul 28 17:36:27.887256 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.887188    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a"} err="failed to get container status \"eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a\": rpc error: code = NotFound desc = could not find container \"eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a\": container with ID starting with eb77ac8605b8a84b2bb54b6179ea3991b9ececa65c6fcb6ea5bdb73f94fb140a not found: ID does not exist"
Jul 28 17:36:27.901215 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.901188    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"]
Jul 28 17:36:27.904628 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:27.904608    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-795b8ljv"]
Jul 28 17:36:28.319517 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:28.319485    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687b42a4-f849-499c-9a1d-78bedb9c673c" path="/var/lib/kubelet/pods/687b42a4-f849-499c-9a1d-78bedb9c673c/volumes"
Jul 28 17:36:42.115283 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.115243    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"]
Jul 28 17:36:42.115743 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.115728    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" containerName="main"
Jul 28 17:36:42.115792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.115745    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" containerName="main"
Jul 28 17:36:42.115792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.115765    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687b42a4-f849-499c-9a1d-78bedb9c673c" containerName="main"
Jul 28 17:36:42.115792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.115771    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="687b42a4-f849-499c-9a1d-78bedb9c673c" containerName="main"
Jul 28 17:36:42.115928 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.115833    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="687b42a4-f849-499c-9a1d-78bedb9c673c" containerName="main"
Jul 28 17:36:42.115928 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.115844    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cefa1bbf-2d38-4d59-b9dd-6d5517d838ad" containerName="main"
Jul 28 17:36:42.120619 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.120593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.124065 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.124041    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 28 17:36:42.124446 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.124376    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-q2tpw\""
Jul 28 17:36:42.125188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.125166    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-74x8n\""
Jul 28 17:36:42.125268 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.125219    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 28 17:36:42.125328 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.125168    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 28 17:36:42.129963 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.129628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c17524d3-00f8-425a-986b-cf0165dcbee6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.129963 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.129687    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.129963 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.129715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862p8\" (UniqueName: \"kubernetes.io/projected/c17524d3-00f8-425a-986b-cf0165dcbee6-kube-api-access-862p8\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.129963 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.129786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.129963 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.129825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.129963 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.129871    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.132267 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.132246    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"]
Jul 28 17:36:42.230991 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.230955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.231211 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.231001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.231211 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.231032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.231211 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.231059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c17524d3-00f8-425a-986b-cf0165dcbee6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.231211 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.231121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.231211 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.231148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862p8\" (UniqueName: \"kubernetes.io/projected/c17524d3-00f8-425a-986b-cf0165dcbee6-kube-api-access-862p8\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.231521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.231403    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-home\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.231521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.231455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.231521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.231488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.233429 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.233409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.233732 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.233710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c17524d3-00f8-425a-986b-cf0165dcbee6-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.239311 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.239286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-862p8\" (UniqueName: \"kubernetes.io/projected/c17524d3-00f8-425a-986b-cf0165dcbee6-kube-api-access-862p8\") pod \"scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.344447 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.344413    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"]
Jul 28 17:36:42.348328 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.348310    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:36:42.351008 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.350988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-946m5\""
Jul 28 17:36:42.357049 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.357025    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"]
Jul 28 17:36:42.432253 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.432162    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:42.433396 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.433373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmwm\" (UniqueName: \"kubernetes.io/projected/6863255c-3cb4-44df-b4a8-92b5e4dc375b-kube-api-access-4tmwm\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9\" (UID: \"6863255c-3cb4-44df-b4a8-92b5e4dc375b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:36:42.433481 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.433469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6863255c-3cb4-44df-b4a8-92b5e4dc375b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9\" (UID: \"6863255c-3cb4-44df-b4a8-92b5e4dc375b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:36:42.534478 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.534044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmwm\" (UniqueName: \"kubernetes.io/projected/6863255c-3cb4-44df-b4a8-92b5e4dc375b-kube-api-access-4tmwm\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9\" (UID: \"6863255c-3cb4-44df-b4a8-92b5e4dc375b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:36:42.534478 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.534224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6863255c-3cb4-44df-b4a8-92b5e4dc375b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9\" (UID: \"6863255c-3cb4-44df-b4a8-92b5e4dc375b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:36:42.537221 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.537190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6863255c-3cb4-44df-b4a8-92b5e4dc375b-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9\" (UID: \"6863255c-3cb4-44df-b4a8-92b5e4dc375b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:36:42.543455 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.543430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmwm\" (UniqueName: \"kubernetes.io/projected/6863255c-3cb4-44df-b4a8-92b5e4dc375b-kube-api-access-4tmwm\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9\" (UID: \"6863255c-3cb4-44df-b4a8-92b5e4dc375b\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:36:42.570679 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.570658    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"]
Jul 28 17:36:42.573003 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:36:42.572982    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17524d3_00f8_425a_986b_cf0165dcbee6.slice/crio-a2419d9fa18ea345b1c833fe3be259b807c29a3e19a5637404be23a34dee9de7 WatchSource:0}: Error finding container a2419d9fa18ea345b1c833fe3be259b807c29a3e19a5637404be23a34dee9de7: Status 404 returned error can't find the container with id a2419d9fa18ea345b1c833fe3be259b807c29a3e19a5637404be23a34dee9de7
Jul 28 17:36:42.659840 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.659807    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:36:42.798452 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.797729    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"]
Jul 28 17:36:42.951263 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.951212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9" event={"ID":"6863255c-3cb4-44df-b4a8-92b5e4dc375b","Type":"ContainerStarted","Data":"dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67"}
Jul 28 17:36:42.951433 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.951279    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9" event={"ID":"6863255c-3cb4-44df-b4a8-92b5e4dc375b","Type":"ContainerStarted","Data":"18535f47e0503fef23a36f7f019e8830917b5e3fab6b711b7f7e91422ca65495"}
Jul 28 17:36:42.951433 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.951403    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:36:42.952413 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.952391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" event={"ID":"c17524d3-00f8-425a-986b-cf0165dcbee6","Type":"ContainerStarted","Data":"a2419d9fa18ea345b1c833fe3be259b807c29a3e19a5637404be23a34dee9de7"}
Jul 28 17:36:42.985027 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:42.984914    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9" podStartSLOduration=0.984898023 podStartE2EDuration="984.898023ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:42.983276942 +0000 UTC m=+1591.212476057" watchObservedRunningTime="2026-07-28 17:36:42.984898023 +0000 UTC m=+1591.214097118"
Jul 28 17:36:44.968166 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:44.968114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" event={"ID":"c17524d3-00f8-425a-986b-cf0165dcbee6","Type":"ContainerStarted","Data":"f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a"}
Jul 28 17:36:44.968654 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:44.968205    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:45.975065 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:45.975026    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" event={"ID":"c17524d3-00f8-425a-986b-cf0165dcbee6","Type":"ContainerStarted","Data":"a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683"}
Jul 28 17:36:46.014923 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:46.014844    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" podStartSLOduration=2.171402027 podStartE2EDuration="4.014824812s" totalImagesPullingTime="1.843422785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:36:46.009980946 +0000 UTC m=+1594.239180042" watchObservedRunningTime="2026-07-28 17:36:46.014824812 +0000 UTC m=+1594.244023907"
Jul 28 17:36:46.432967 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:46.432917    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:52.432509 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:52.432473    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:52.444593 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:52.444569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:53.020336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:53.020306    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:36:54.026953 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:36:54.026924    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:37:13.962656 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:37:13.962623    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:39:19.892575 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:19.892544    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"]
Jul 28 17:39:19.903821 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:39:19.903783    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:39:19.903821 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:39:19.903821 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:39:19.903821 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683" cmd=["/bin/sleep","15"]
Jul 28 17:39:19.904108 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:19.903845    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" message="" err=<
Jul 28 17:39:19.904108 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:39:19.904108 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:39:19.904108 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:39:19.904108 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:39:19.903859    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:39:19.904108 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:39:19.904108 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:39:19.904108 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerName="main" containerID="cri-o://a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683"
Jul 28 17:39:19.904108 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:19.903906    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerName="main" containerID="cri-o://a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683" gracePeriod=60
Jul 28 17:39:19.909616 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:19.909587    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"]
Jul 28 17:39:20.009526 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.009488    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.45:8001/health\": dial tcp 10.132.0.45:8001: connect: connection refused"
Jul 28 17:39:20.114553 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.114524    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerName="llm-d-routing-sidecar" containerID="cri-o://f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a" gracePeriod=60
Jul 28 17:39:20.367676 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.367650    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:39:20.456218 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.456189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-862p8\" (UniqueName: \"kubernetes.io/projected/c17524d3-00f8-425a-986b-cf0165dcbee6-kube-api-access-862p8\") pod \"c17524d3-00f8-425a-986b-cf0165dcbee6\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") "
Jul 28 17:39:20.456398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.456231    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-tmp-dir\") pod \"c17524d3-00f8-425a-986b-cf0165dcbee6\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") "
Jul 28 17:39:20.456398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.456251    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-home\") pod \"c17524d3-00f8-425a-986b-cf0165dcbee6\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") "
Jul 28 17:39:20.456398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.456269    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-model-cache\") pod \"c17524d3-00f8-425a-986b-cf0165dcbee6\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") "
Jul 28 17:39:20.456398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.456301    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-dshm\") pod \"c17524d3-00f8-425a-986b-cf0165dcbee6\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") "
Jul 28 17:39:20.456398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.456326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c17524d3-00f8-425a-986b-cf0165dcbee6-tls-certs\") pod \"c17524d3-00f8-425a-986b-cf0165dcbee6\" (UID: \"c17524d3-00f8-425a-986b-cf0165dcbee6\") "
Jul 28 17:39:20.456676 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.456503    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-home" (OuterVolumeSpecName: "home") pod "c17524d3-00f8-425a-986b-cf0165dcbee6" (UID: "c17524d3-00f8-425a-986b-cf0165dcbee6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.456676 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.456528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c17524d3-00f8-425a-986b-cf0165dcbee6" (UID: "c17524d3-00f8-425a-986b-cf0165dcbee6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.456676 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.456559    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-model-cache" (OuterVolumeSpecName: "model-cache") pod "c17524d3-00f8-425a-986b-cf0165dcbee6" (UID: "c17524d3-00f8-425a-986b-cf0165dcbee6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.456676 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.456601    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.458543 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.458516    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-dshm" (OuterVolumeSpecName: "dshm") pod "c17524d3-00f8-425a-986b-cf0165dcbee6" (UID: "c17524d3-00f8-425a-986b-cf0165dcbee6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:39:20.458657 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.458583    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c17524d3-00f8-425a-986b-cf0165dcbee6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c17524d3-00f8-425a-986b-cf0165dcbee6" (UID: "c17524d3-00f8-425a-986b-cf0165dcbee6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:39:20.458657 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.458617    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17524d3-00f8-425a-986b-cf0165dcbee6-kube-api-access-862p8" (OuterVolumeSpecName: "kube-api-access-862p8") pod "c17524d3-00f8-425a-986b-cf0165dcbee6" (UID: "c17524d3-00f8-425a-986b-cf0165dcbee6"). InnerVolumeSpecName "kube-api-access-862p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:39:20.557362 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.557281    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-862p8\" (UniqueName: \"kubernetes.io/projected/c17524d3-00f8-425a-986b-cf0165dcbee6-kube-api-access-862p8\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.557362 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.557307    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.557362 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.557317    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.557362 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.557325    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c17524d3-00f8-425a-986b-cf0165dcbee6-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.557362 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.557335    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c17524d3-00f8-425a-986b-cf0165dcbee6-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:39:20.725842 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.725810    2579 generic.go:356] "Generic (PLEG): container finished" podID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerID="a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683" exitCode=0
Jul 28 17:39:20.725842 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.725834    2579 generic.go:356] "Generic (PLEG): container finished" podID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerID="f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a" exitCode=0
Jul 28 17:39:20.726037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.725858    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"
Jul 28 17:39:20.726037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.725891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" event={"ID":"c17524d3-00f8-425a-986b-cf0165dcbee6","Type":"ContainerDied","Data":"a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683"}
Jul 28 17:39:20.726037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.725937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" event={"ID":"c17524d3-00f8-425a-986b-cf0165dcbee6","Type":"ContainerDied","Data":"f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a"}
Jul 28 17:39:20.726037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.725956    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n" event={"ID":"c17524d3-00f8-425a-986b-cf0165dcbee6","Type":"ContainerDied","Data":"a2419d9fa18ea345b1c833fe3be259b807c29a3e19a5637404be23a34dee9de7"}
Jul 28 17:39:20.726037 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.725976    2579 scope.go:117] "RemoveContainer" containerID="a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683"
Jul 28 17:39:20.735207 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.735187    2579 scope.go:117] "RemoveContainer" containerID="f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a"
Jul 28 17:39:20.742947 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.742925    2579 scope.go:117] "RemoveContainer" containerID="a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683"
Jul 28 17:39:20.743211 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:39:20.743193    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683\": container with ID starting with a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683 not found: ID does not exist" containerID="a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683"
Jul 28 17:39:20.743279 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.743219    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683"} err="failed to get container status \"a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683\": rpc error: code = NotFound desc = could not find container \"a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683\": container with ID starting with a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683 not found: ID does not exist"
Jul 28 17:39:20.743279 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.743236    2579 scope.go:117] "RemoveContainer" containerID="f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a"
Jul 28 17:39:20.743501 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:39:20.743482    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a\": container with ID starting with f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a not found: ID does not exist" containerID="f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a"
Jul 28 17:39:20.743535 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.743507    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a"} err="failed to get container status \"f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a\": rpc error: code = NotFound desc = could not find container \"f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a\": container with ID starting with f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a not found: ID does not exist"
Jul 28 17:39:20.743535 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.743523    2579 scope.go:117] "RemoveContainer" containerID="a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683"
Jul 28 17:39:20.743726 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.743704    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683"} err="failed to get container status \"a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683\": rpc error: code = NotFound desc = could not find container \"a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683\": container with ID starting with a2eb2d1358567e9a9cf3b77dfade2d1c6b04ebcc4467cf4e977cbca8437ec683 not found: ID does not exist"
Jul 28 17:39:20.743767 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.743727    2579 scope.go:117] "RemoveContainer" containerID="f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a"
Jul 28 17:39:20.743917 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.743902    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a"} err="failed to get container status \"f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a\": rpc error: code = NotFound desc = could not find container \"f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a\": container with ID starting with f223772976a39252e3e1ff4b523468cd753b1da0bf3a3c9cee36b8f0663ce91a not found: ID does not exist"
Jul 28 17:39:20.749267 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.749247    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"]
Jul 28 17:39:20.753044 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:20.753024    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-6c4d7b4898-c6k6n"]
Jul 28 17:39:22.320612 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:22.320578    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" path="/var/lib/kubelet/pods/c17524d3-00f8-425a-986b-cf0165dcbee6/volumes"
Jul 28 17:39:23.963196 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:23.963158    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9" podUID="6863255c-3cb4-44df-b4a8-92b5e4dc375b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:39:33.962274 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:33.962235    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9" podUID="6863255c-3cb4-44df-b4a8-92b5e4dc375b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:39:34.929278 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:34.929238    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9" podUID="6863255c-3cb4-44df-b4a8-92b5e4dc375b" containerName="main" containerID="cri-o://dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67" gracePeriod=45
Jul 28 17:39:35.182844 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.182792    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:39:35.291240 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.291203    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tmwm\" (UniqueName: \"kubernetes.io/projected/6863255c-3cb4-44df-b4a8-92b5e4dc375b-kube-api-access-4tmwm\") pod \"6863255c-3cb4-44df-b4a8-92b5e4dc375b\" (UID: \"6863255c-3cb4-44df-b4a8-92b5e4dc375b\") "
Jul 28 17:39:35.291441 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.291254    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6863255c-3cb4-44df-b4a8-92b5e4dc375b-tls-certs\") pod \"6863255c-3cb4-44df-b4a8-92b5e4dc375b\" (UID: \"6863255c-3cb4-44df-b4a8-92b5e4dc375b\") "
Jul 28 17:39:35.293308 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.293277    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6863255c-3cb4-44df-b4a8-92b5e4dc375b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6863255c-3cb4-44df-b4a8-92b5e4dc375b" (UID: "6863255c-3cb4-44df-b4a8-92b5e4dc375b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:39:35.293415 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.293323    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6863255c-3cb4-44df-b4a8-92b5e4dc375b-kube-api-access-4tmwm" (OuterVolumeSpecName: "kube-api-access-4tmwm") pod "6863255c-3cb4-44df-b4a8-92b5e4dc375b" (UID: "6863255c-3cb4-44df-b4a8-92b5e4dc375b"). InnerVolumeSpecName "kube-api-access-4tmwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:39:35.392308 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.392269    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmwm\" (UniqueName: \"kubernetes.io/projected/6863255c-3cb4-44df-b4a8-92b5e4dc375b-kube-api-access-4tmwm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:39:35.392308 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.392305    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6863255c-3cb4-44df-b4a8-92b5e4dc375b-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:39:35.804000 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.803963    2579 generic.go:356] "Generic (PLEG): container finished" podID="6863255c-3cb4-44df-b4a8-92b5e4dc375b" containerID="dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67" exitCode=0
Jul 28 17:39:35.804198 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.804012    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"
Jul 28 17:39:35.804198 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.804046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9" event={"ID":"6863255c-3cb4-44df-b4a8-92b5e4dc375b","Type":"ContainerDied","Data":"dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67"}
Jul 28 17:39:35.804198 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.804123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9" event={"ID":"6863255c-3cb4-44df-b4a8-92b5e4dc375b","Type":"ContainerDied","Data":"18535f47e0503fef23a36f7f019e8830917b5e3fab6b711b7f7e91422ca65495"}
Jul 28 17:39:35.804198 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.804150    2579 scope.go:117] "RemoveContainer" containerID="dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67"
Jul 28 17:39:35.813383 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.813365    2579 scope.go:117] "RemoveContainer" containerID="dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67"
Jul 28 17:39:35.813631 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:39:35.813613    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67\": container with ID starting with dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67 not found: ID does not exist" containerID="dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67"
Jul 28 17:39:35.813697 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.813644    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67"} err="failed to get container status \"dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67\": rpc error: code = NotFound desc = could not find container \"dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67\": container with ID starting with dc541ec20877cd8499f78b5d137da41f39e5a882f5bbc2c1eb72512cd1cf8d67 not found: ID does not exist"
Jul 28 17:39:35.826002 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.825978    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"]
Jul 28 17:39:35.829885 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:35.829864    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-f9tcln9"]
Jul 28 17:39:36.325697 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:36.320465    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6863255c-3cb4-44df-b4a8-92b5e4dc375b" path="/var/lib/kubelet/pods/6863255c-3cb4-44df-b4a8-92b5e4dc375b/volumes"
Jul 28 17:39:47.203226 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.203193    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"]
Jul 28 17:39:47.203747 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.203728    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerName="llm-d-routing-sidecar"
Jul 28 17:39:47.203831 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.203750    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerName="llm-d-routing-sidecar"
Jul 28 17:39:47.203831 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.203767    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6863255c-3cb4-44df-b4a8-92b5e4dc375b" containerName="main"
Jul 28 17:39:47.203831 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.203775    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6863255c-3cb4-44df-b4a8-92b5e4dc375b" containerName="main"
Jul 28 17:39:47.203831 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.203788    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerName="main"
Jul 28 17:39:47.203831 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.203797    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerName="main"
Jul 28 17:39:47.204102 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.203911    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerName="llm-d-routing-sidecar"
Jul 28 17:39:47.204102 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.203928    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c17524d3-00f8-425a-986b-cf0165dcbee6" containerName="main"
Jul 28 17:39:47.204102 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.203941    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6863255c-3cb4-44df-b4a8-92b5e4dc375b" containerName="main"
Jul 28 17:39:47.207193 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.207171    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.211593 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.211569    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 28 17:39:47.212793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.212774    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 28 17:39:47.212793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.212786    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-pcqc7\""
Jul 28 17:39:47.217048 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.217030    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-94zsd\""
Jul 28 17:39:47.217189 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.217168    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 28 17:39:47.226193 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.226172    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"]
Jul 28 17:39:47.296328 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.296287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.296328 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.296336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.296605 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.296384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.296605 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.296407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6jfq\" (UniqueName: \"kubernetes.io/projected/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-kube-api-access-z6jfq\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.296605 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.296480    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.296605 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.296516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.397150 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.397109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.397150 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.397156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.397425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.397184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6jfq\" (UniqueName: \"kubernetes.io/projected/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-kube-api-access-z6jfq\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.397425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.397239    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.397425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.397269    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.397425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.397331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.397681 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.397661    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.397751 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.397697    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.397751 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.397740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.399470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.399445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.399778 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.399761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.408207 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.408181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6jfq\" (UniqueName: \"kubernetes.io/projected/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-kube-api-access-z6jfq\") pod \"scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.434655 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.434614    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"]
Jul 28 17:39:47.438580 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.438554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:39:47.441351 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.441331    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-gb9fc\""
Jul 28 17:39:47.447934 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.447910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"]
Jul 28 17:39:47.516761 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.516663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.599203 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.599168    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n2wf\" (UniqueName: \"kubernetes.io/projected/0f200bf3-6f54-406c-8875-4fc870df170d-kube-api-access-5n2wf\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr\" (UID: \"0f200bf3-6f54-406c-8875-4fc870df170d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:39:47.599432 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.599223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f200bf3-6f54-406c-8875-4fc870df170d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr\" (UID: \"0f200bf3-6f54-406c-8875-4fc870df170d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:39:47.660291 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.660252    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"]
Jul 28 17:39:47.699907 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.699880    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n2wf\" (UniqueName: \"kubernetes.io/projected/0f200bf3-6f54-406c-8875-4fc870df170d-kube-api-access-5n2wf\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr\" (UID: \"0f200bf3-6f54-406c-8875-4fc870df170d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:39:47.700025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.699920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f200bf3-6f54-406c-8875-4fc870df170d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr\" (UID: \"0f200bf3-6f54-406c-8875-4fc870df170d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:39:47.702463 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.702442    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f200bf3-6f54-406c-8875-4fc870df170d-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr\" (UID: \"0f200bf3-6f54-406c-8875-4fc870df170d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:39:47.707993 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.707962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n2wf\" (UniqueName: \"kubernetes.io/projected/0f200bf3-6f54-406c-8875-4fc870df170d-kube-api-access-5n2wf\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr\" (UID: \"0f200bf3-6f54-406c-8875-4fc870df170d\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:39:47.751463 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.751436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:39:47.865782 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.865734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" event={"ID":"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73","Type":"ContainerStarted","Data":"83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d"}
Jul 28 17:39:47.865946 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.865798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" event={"ID":"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73","Type":"ContainerStarted","Data":"6204801a8ea83563c995f859afe34ed139a92c2db8c9c1b1455170ada0af7854"}
Jul 28 17:39:47.865946 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.865893    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:47.881891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:47.881846    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"]
Jul 28 17:39:47.885475 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:39:47.885448    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f200bf3_6f54_406c_8875_4fc870df170d.slice/crio-61a84f71b42354428213226a03f62e2fae4c4beb45d1240fc40531c786a69783 WatchSource:0}: Error finding container 61a84f71b42354428213226a03f62e2fae4c4beb45d1240fc40531c786a69783: Status 404 returned error can't find the container with id 61a84f71b42354428213226a03f62e2fae4c4beb45d1240fc40531c786a69783
Jul 28 17:39:48.872700 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:48.872643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" event={"ID":"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73","Type":"ContainerStarted","Data":"adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d"}
Jul 28 17:39:48.874060 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:48.874035    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr" event={"ID":"0f200bf3-6f54-406c-8875-4fc870df170d","Type":"ContainerStarted","Data":"a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9"}
Jul 28 17:39:48.874166 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:48.874100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr" event={"ID":"0f200bf3-6f54-406c-8875-4fc870df170d","Type":"ContainerStarted","Data":"61a84f71b42354428213226a03f62e2fae4c4beb45d1240fc40531c786a69783"}
Jul 28 17:39:48.874166 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:48.874125    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:39:48.893336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:48.893286    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" podStartSLOduration=1.8932677359999999 podStartE2EDuration="1.893267736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:48.892243381 +0000 UTC m=+1777.121442507" watchObservedRunningTime="2026-07-28 17:39:48.893267736 +0000 UTC m=+1777.122466833"
Jul 28 17:39:48.908498 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:48.908451    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr" podStartSLOduration=1.908433322 podStartE2EDuration="1.908433322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:48.907184085 +0000 UTC m=+1777.136383206" watchObservedRunningTime="2026-07-28 17:39:48.908433322 +0000 UTC m=+1777.137632423"
Jul 28 17:39:49.517510 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:49.517470    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:57.517182 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:57.517147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:57.529294 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:57.529271    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:57.531840 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:57.531818    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:39:57.932744 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:39:57.932717    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:40:12.329509 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:12.329475    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:40:12.333595 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:12.333565    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:40:19.880875 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:19.880844    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:40:21.524329 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:21.524285    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"]
Jul 28 17:40:21.526547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:21.526522    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"]
Jul 28 17:40:21.533733 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:40:21.533691    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:40:21.533733 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:40:21.533733 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:40:21.533733 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d" cmd=["/bin/sleep","15"]
Jul 28 17:40:21.533983 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:21.533763    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" message="" err=<
Jul 28 17:40:21.533983 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:40:21.533983 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:40:21.533983 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:40:21.533983 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:40:21.533779    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:40:21.533983 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:40:21.533983 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:40:21.533983 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerName="main" containerID="cri-o://adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d"
Jul 28 17:40:21.533983 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:21.533837    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerName="main" containerID="cri-o://adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d" gracePeriod=60
Jul 28 17:40:21.750028 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:21.749997    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerName="llm-d-routing-sidecar" containerID="cri-o://83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d" gracePeriod=60
Jul 28 17:40:22.003319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.003291    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:40:22.051830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.051796    2579 generic.go:356] "Generic (PLEG): container finished" podID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerID="adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d" exitCode=0
Jul 28 17:40:22.051830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.051829    2579 generic.go:356] "Generic (PLEG): container finished" podID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerID="83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d" exitCode=0
Jul 28 17:40:22.052031 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.051844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" event={"ID":"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73","Type":"ContainerDied","Data":"adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d"}
Jul 28 17:40:22.052031 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.051901    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" event={"ID":"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73","Type":"ContainerDied","Data":"83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d"}
Jul 28 17:40:22.052031 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.051918    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" event={"ID":"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73","Type":"ContainerDied","Data":"6204801a8ea83563c995f859afe34ed139a92c2db8c9c1b1455170ada0af7854"}
Jul 28 17:40:22.052031 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.051854    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"
Jul 28 17:40:22.052031 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.051939    2579 scope.go:117] "RemoveContainer" containerID="adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d"
Jul 28 17:40:22.060615 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.060600    2579 scope.go:117] "RemoveContainer" containerID="83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d"
Jul 28 17:40:22.068251 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.068235    2579 scope.go:117] "RemoveContainer" containerID="adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d"
Jul 28 17:40:22.068508 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:40:22.068492    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d\": container with ID starting with adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d not found: ID does not exist" containerID="adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d"
Jul 28 17:40:22.068581 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.068518    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d"} err="failed to get container status \"adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d\": rpc error: code = NotFound desc = could not find container \"adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d\": container with ID starting with adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d not found: ID does not exist"
Jul 28 17:40:22.068581 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.068540    2579 scope.go:117] "RemoveContainer" containerID="83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d"
Jul 28 17:40:22.068763 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:40:22.068746    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d\": container with ID starting with 83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d not found: ID does not exist" containerID="83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d"
Jul 28 17:40:22.068803 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.068770    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d"} err="failed to get container status \"83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d\": rpc error: code = NotFound desc = could not find container \"83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d\": container with ID starting with 83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d not found: ID does not exist"
Jul 28 17:40:22.068803 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.068786    2579 scope.go:117] "RemoveContainer" containerID="adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d"
Jul 28 17:40:22.069000 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.068985    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d"} err="failed to get container status \"adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d\": rpc error: code = NotFound desc = could not find container \"adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d\": container with ID starting with adc5ef8db3f3a645549d4a226169672cb2700e66a86c9c846082514359e21a3d not found: ID does not exist"
Jul 28 17:40:22.069046 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.069001    2579 scope.go:117] "RemoveContainer" containerID="83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d"
Jul 28 17:40:22.069216 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.069196    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d"} err="failed to get container status \"83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d\": rpc error: code = NotFound desc = could not find container \"83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d\": container with ID starting with 83305542514ce647560ff0c719e773d4faee68bdbf23b551289aafa6cd57d54d not found: ID does not exist"
Jul 28 17:40:22.107849 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.107815    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-dshm\") pod \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") "
Jul 28 17:40:22.108016 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.107856    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-home\") pod \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") "
Jul 28 17:40:22.108016 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.107917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6jfq\" (UniqueName: \"kubernetes.io/projected/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-kube-api-access-z6jfq\") pod \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") "
Jul 28 17:40:22.108016 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.107942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tmp-dir\") pod \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") "
Jul 28 17:40:22.108016 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.107992    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tls-certs\") pod \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") "
Jul 28 17:40:22.108264 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.108020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-model-cache\") pod \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\" (UID: \"b17255b8-42c4-4cd7-8882-1c1bf8fd1f73\") "
Jul 28 17:40:22.108264 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.108232    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-home" (OuterVolumeSpecName: "home") pod "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" (UID: "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:22.108372 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.108276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" (UID: "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:22.108422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.108382    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:40:22.108422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.108386    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-model-cache" (OuterVolumeSpecName: "model-cache") pod "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" (UID: "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:22.108422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.108395    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:40:22.110091 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.110045    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-dshm" (OuterVolumeSpecName: "dshm") pod "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" (UID: "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:22.110200 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.110154    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-kube-api-access-z6jfq" (OuterVolumeSpecName: "kube-api-access-z6jfq") pod "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" (UID: "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73"). InnerVolumeSpecName "kube-api-access-z6jfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:22.110265 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.110236    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" (UID: "b17255b8-42c4-4cd7-8882-1c1bf8fd1f73"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:40:22.209353 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.209270    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:40:22.209353 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.209299    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:40:22.209353 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.209307    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:40:22.209353 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.209315    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6jfq\" (UniqueName: \"kubernetes.io/projected/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73-kube-api-access-z6jfq\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:40:22.368914 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.368880    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"]
Jul 28 17:40:22.373676 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.373649    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp"]
Jul 28 17:40:22.922223 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:22.922180    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-5d7b9b9c45-hbqvp" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8001/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:40:24.320138 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:24.320104    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" path="/var/lib/kubelet/pods/b17255b8-42c4-4cd7-8882-1c1bf8fd1f73/volumes"
Jul 28 17:40:29.881186 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:29.881147    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr" podUID="0f200bf3-6f54-406c-8875-4fc870df170d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:40:36.545531 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:36.545489    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr" podUID="0f200bf3-6f54-406c-8875-4fc870df170d" containerName="main" containerID="cri-o://a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9" gracePeriod=45
Jul 28 17:40:36.804222 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:36.804162    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:40:36.941817 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:36.941787    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f200bf3-6f54-406c-8875-4fc870df170d-tls-certs\") pod \"0f200bf3-6f54-406c-8875-4fc870df170d\" (UID: \"0f200bf3-6f54-406c-8875-4fc870df170d\") "
Jul 28 17:40:36.941982 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:36.941880    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n2wf\" (UniqueName: \"kubernetes.io/projected/0f200bf3-6f54-406c-8875-4fc870df170d-kube-api-access-5n2wf\") pod \"0f200bf3-6f54-406c-8875-4fc870df170d\" (UID: \"0f200bf3-6f54-406c-8875-4fc870df170d\") "
Jul 28 17:40:36.943862 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:36.943829    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f200bf3-6f54-406c-8875-4fc870df170d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f200bf3-6f54-406c-8875-4fc870df170d" (UID: "0f200bf3-6f54-406c-8875-4fc870df170d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:40:36.943862 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:36.943854    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f200bf3-6f54-406c-8875-4fc870df170d-kube-api-access-5n2wf" (OuterVolumeSpecName: "kube-api-access-5n2wf") pod "0f200bf3-6f54-406c-8875-4fc870df170d" (UID: "0f200bf3-6f54-406c-8875-4fc870df170d"). InnerVolumeSpecName "kube-api-access-5n2wf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:37.043000 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.042961    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f200bf3-6f54-406c-8875-4fc870df170d-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:40:37.043000 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.042996    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n2wf\" (UniqueName: \"kubernetes.io/projected/0f200bf3-6f54-406c-8875-4fc870df170d-kube-api-access-5n2wf\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:40:37.128970 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.128929    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f200bf3-6f54-406c-8875-4fc870df170d" containerID="a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9" exitCode=0
Jul 28 17:40:37.129161 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.128979    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"
Jul 28 17:40:37.129161 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.129017    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr" event={"ID":"0f200bf3-6f54-406c-8875-4fc870df170d","Type":"ContainerDied","Data":"a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9"}
Jul 28 17:40:37.129161 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.129073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr" event={"ID":"0f200bf3-6f54-406c-8875-4fc870df170d","Type":"ContainerDied","Data":"61a84f71b42354428213226a03f62e2fae4c4beb45d1240fc40531c786a69783"}
Jul 28 17:40:37.129161 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.129117    2579 scope.go:117] "RemoveContainer" containerID="a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9"
Jul 28 17:40:37.137836 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.137817    2579 scope.go:117] "RemoveContainer" containerID="a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9"
Jul 28 17:40:37.138135 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:40:37.138111    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9\": container with ID starting with a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9 not found: ID does not exist" containerID="a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9"
Jul 28 17:40:37.138214 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.138143    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9"} err="failed to get container status \"a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9\": rpc error: code = NotFound desc = could not find container \"a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9\": container with ID starting with a067fc44f14302de0ef0e010050077f089e9bded18ec765bf62bbd67403517e9 not found: ID does not exist"
Jul 28 17:40:37.151949 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.151919    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"]
Jul 28 17:40:37.155940 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:37.155917    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedrlwgr"]
Jul 28 17:40:38.320015 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:38.319979    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f200bf3-6f54-406c-8875-4fc870df170d" path="/var/lib/kubelet/pods/0f200bf3-6f54-406c-8875-4fc870df170d/volumes"
Jul 28 17:40:45.662426 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.662390    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"]
Jul 28 17:40:45.662804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.662751    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f200bf3-6f54-406c-8875-4fc870df170d" containerName="main"
Jul 28 17:40:45.662804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.662761    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f200bf3-6f54-406c-8875-4fc870df170d" containerName="main"
Jul 28 17:40:45.662804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.662779    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerName="llm-d-routing-sidecar"
Jul 28 17:40:45.662804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.662785    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerName="llm-d-routing-sidecar"
Jul 28 17:40:45.662804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.662799    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerName="main"
Jul 28 17:40:45.662804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.662805    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerName="main"
Jul 28 17:40:45.662999 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.662859    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f200bf3-6f54-406c-8875-4fc870df170d" containerName="main"
Jul 28 17:40:45.662999 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.662867    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerName="llm-d-routing-sidecar"
Jul 28 17:40:45.662999 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.662873    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b17255b8-42c4-4cd7-8882-1c1bf8fd1f73" containerName="main"
Jul 28 17:40:45.665957 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.665937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.668744 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.668718    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 28 17:40:45.668855 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.668770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 28 17:40:45.668855 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.668775    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 28 17:40:45.670064 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.670045    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-h6ncf\""
Jul 28 17:40:45.670183 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.670106    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 28 17:40:45.678487 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.678464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"]
Jul 28 17:40:45.827887 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.827844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.828043 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.827895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.828043 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.827993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ad712-3573-448c-bab1-c03724bd085b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.828043 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.828036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.828262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.828058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.828262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.828073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rvw2\" (UniqueName: \"kubernetes.io/projected/d20ad712-3573-448c-bab1-c03724bd085b-kube-api-access-8rvw2\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.828262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.828193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.896344 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.896309    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"]
Jul 28 17:40:45.899879 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.899857    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:45.910360 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.910333    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"]
Jul 28 17:40:45.929681 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.929610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.929681 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.929650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.929879 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.929690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ad712-3573-448c-bab1-c03724bd085b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.929879 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.929727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.929879 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.929751    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.929879 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.929778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rvw2\" (UniqueName: \"kubernetes.io/projected/d20ad712-3573-448c-bab1-c03724bd085b-kube-api-access-8rvw2\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.929879 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.929826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.930794 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.930239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.931200 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.931172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.931336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.931201    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-home\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.931336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.931312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-model-cache\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.932267 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.932246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-dshm\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.932885 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.932865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ad712-3573-448c-bab1-c03724bd085b-tls-certs\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.941182 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.941158    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rvw2\" (UniqueName: \"kubernetes.io/projected/d20ad712-3573-448c-bab1-c03724bd085b-kube-api-access-8rvw2\") pod \"tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:45.977149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:45.977117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:46.031477 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.031440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:46.031640 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.031518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:46.031640 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.031568    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:46.115555 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.115531    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"]
Jul 28 17:40:46.117796 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:40:46.117766    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20ad712_3573_448c_bab1_c03724bd085b.slice/crio-954eb5c0e02c655c279d9040a751949e5a69f83dc859f18730df1d69ee805daa WatchSource:0}: Error finding container 954eb5c0e02c655c279d9040a751949e5a69f83dc859f18730df1d69ee805daa: Status 404 returned error can't find the container with id 954eb5c0e02c655c279d9040a751949e5a69f83dc859f18730df1d69ee805daa
Jul 28 17:40:46.132674 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.132652    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:46.132792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.132694    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:46.132792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.132720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:46.133134 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.133108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tmp-dir\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:46.133262 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.133115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-kserve-provision-location\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:46.135161 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.135136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tls-certs\") pod \"tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:46.176383 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.176344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r" event={"ID":"d20ad712-3573-448c-bab1-c03724bd085b","Type":"ContainerStarted","Data":"e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e"}
Jul 28 17:40:46.176477 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.176408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r" event={"ID":"d20ad712-3573-448c-bab1-c03724bd085b","Type":"ContainerStarted","Data":"954eb5c0e02c655c279d9040a751949e5a69f83dc859f18730df1d69ee805daa"}
Jul 28 17:40:46.212184 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.212097    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:40:46.352336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:46.352288    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"]
Jul 28 17:40:46.354529 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:40:46.354501    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1494cd_02cd_4cf2_80ac_5ba00490a923.slice/crio-1593c5b51b6b726cc22c5fa30e99dc699858b432d3f1bba5fb624f37bf99dd5a WatchSource:0}: Error finding container 1593c5b51b6b726cc22c5fa30e99dc699858b432d3f1bba5fb624f37bf99dd5a: Status 404 returned error can't find the container with id 1593c5b51b6b726cc22c5fa30e99dc699858b432d3f1bba5fb624f37bf99dd5a
Jul 28 17:40:47.183017 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:47.182983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd" event={"ID":"dd1494cd-02cd-4cf2-80ac-5ba00490a923","Type":"ContainerStarted","Data":"62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c"}
Jul 28 17:40:47.183552 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:47.183046    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd" event={"ID":"dd1494cd-02cd-4cf2-80ac-5ba00490a923","Type":"ContainerStarted","Data":"1593c5b51b6b726cc22c5fa30e99dc699858b432d3f1bba5fb624f37bf99dd5a"}
Jul 28 17:40:53.220114 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:53.220064    2579 generic.go:356] "Generic (PLEG): container finished" podID="d20ad712-3573-448c-bab1-c03724bd085b" containerID="e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e" exitCode=0
Jul 28 17:40:53.220535 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:53.220110    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r" event={"ID":"d20ad712-3573-448c-bab1-c03724bd085b","Type":"ContainerDied","Data":"e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e"}
Jul 28 17:40:53.221680 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:53.221661    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd1494cd-02cd-4cf2-80ac-5ba00490a923" containerID="62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c" exitCode=0
Jul 28 17:40:53.221758 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:53.221691    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd" event={"ID":"dd1494cd-02cd-4cf2-80ac-5ba00490a923","Type":"ContainerDied","Data":"62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c"}
Jul 28 17:40:53.222603 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:53.222589    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:40:54.231560 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:54.231521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r" event={"ID":"d20ad712-3573-448c-bab1-c03724bd085b","Type":"ContainerStarted","Data":"b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c"}
Jul 28 17:40:54.254223 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:54.253605    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r" podStartSLOduration=2.984126036 podStartE2EDuration="9.253584416s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.26945838s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:40:54.251651613 +0000 UTC m=+1842.480850711" watchObservedRunningTime="2026-07-28 17:40:54.253584416 +0000 UTC m=+1842.482783514"
Jul 28 17:40:54.978149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:54.978112    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:55.977234 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:55.977199    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:55.990733 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:55.990710    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:40:56.259050 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:40:56.258962    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:41:33.456528 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:41:33.456482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd" event={"ID":"dd1494cd-02cd-4cf2-80ac-5ba00490a923","Type":"ContainerStarted","Data":"57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf"}
Jul 28 17:41:33.457410 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:41:33.457386    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:41:33.477228 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:41:33.477175    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd" podStartSLOduration=2.339358787 podStartE2EDuration="48.477156101s" totalImagesPullingTime="39.414881081s" totalInitContainerRuntime="6.722916233s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:41:33.475832112 +0000 UTC m=+1881.705031227" watchObservedRunningTime="2026-07-28 17:41:33.477156101 +0000 UTC m=+1881.706355201"
Jul 28 17:42:04.472293 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:04.472257    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:42:06.261948 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:06.261918    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-cf47448b_tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd_dd1494cd-02cd-4cf2-80ac-5ba00490a923/main/0.log"
Jul 28 17:42:06.662141 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:06.662104    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"]
Jul 28 17:42:06.662914 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:06.662412    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd" podUID="dd1494cd-02cd-4cf2-80ac-5ba00490a923" containerName="main" containerID="cri-o://57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf" gracePeriod=30
Jul 28 17:42:06.675340 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:06.675305    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"]
Jul 28 17:42:06.686942 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:42:06.686905    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:42:06.686942 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:42:06.686942 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:42:06.686942 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c" cmd=["/bin/sleep","15"]
Jul 28 17:42:06.687313 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:06.686960    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r" message="" err=<
Jul 28 17:42:06.687313 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:42:06.687313 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:42:06.687313 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:42:06.687313 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:42:06.686975    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:42:06.687313 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:42:06.687313 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:42:06.687313 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r" podUID="d20ad712-3573-448c-bab1-c03724bd085b" containerName="main" containerID="cri-o://b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c"
Jul 28 17:42:06.687313 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:06.687014    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r" podUID="d20ad712-3573-448c-bab1-c03724bd085b" containerName="main" containerID="cri-o://b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c" gracePeriod=60
Jul 28 17:42:07.053280 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.053254    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:42:07.119875 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.119715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-kserve-provision-location\") pod \"d20ad712-3573-448c-bab1-c03724bd085b\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") "
Jul 28 17:42:07.119875 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.119753    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-home\") pod \"d20ad712-3573-448c-bab1-c03724bd085b\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") "
Jul 28 17:42:07.119875 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.119812    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-tmp-dir\") pod \"d20ad712-3573-448c-bab1-c03724bd085b\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") "
Jul 28 17:42:07.119875 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.119858    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-dshm\") pod \"d20ad712-3573-448c-bab1-c03724bd085b\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") "
Jul 28 17:42:07.120152 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.119886    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ad712-3573-448c-bab1-c03724bd085b-tls-certs\") pod \"d20ad712-3573-448c-bab1-c03724bd085b\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") "
Jul 28 17:42:07.120152 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.119923    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-model-cache\") pod \"d20ad712-3573-448c-bab1-c03724bd085b\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") "
Jul 28 17:42:07.120152 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.119965    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rvw2\" (UniqueName: \"kubernetes.io/projected/d20ad712-3573-448c-bab1-c03724bd085b-kube-api-access-8rvw2\") pod \"d20ad712-3573-448c-bab1-c03724bd085b\" (UID: \"d20ad712-3573-448c-bab1-c03724bd085b\") "
Jul 28 17:42:07.120152 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.120020    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-home" (OuterVolumeSpecName: "home") pod "d20ad712-3573-448c-bab1-c03724bd085b" (UID: "d20ad712-3573-448c-bab1-c03724bd085b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.120360 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.120267    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.120832 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.120795    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d20ad712-3573-448c-bab1-c03724bd085b" (UID: "d20ad712-3573-448c-bab1-c03724bd085b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.120943 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.120855    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d20ad712-3573-448c-bab1-c03724bd085b" (UID: "d20ad712-3573-448c-bab1-c03724bd085b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.122979 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.122954    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-dshm" (OuterVolumeSpecName: "dshm") pod "d20ad712-3573-448c-bab1-c03724bd085b" (UID: "d20ad712-3573-448c-bab1-c03724bd085b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.122979 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.122957    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d20ad712-3573-448c-bab1-c03724bd085b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d20ad712-3573-448c-bab1-c03724bd085b" (UID: "d20ad712-3573-448c-bab1-c03724bd085b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:07.123303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.123276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20ad712-3573-448c-bab1-c03724bd085b-kube-api-access-8rvw2" (OuterVolumeSpecName: "kube-api-access-8rvw2") pod "d20ad712-3573-448c-bab1-c03724bd085b" (UID: "d20ad712-3573-448c-bab1-c03724bd085b"). InnerVolumeSpecName "kube-api-access-8rvw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:42:07.179630 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.179603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d20ad712-3573-448c-bab1-c03724bd085b" (UID: "d20ad712-3573-448c-bab1-c03724bd085b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.221025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.220946    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.221025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.220975    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.221025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.220990    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.221025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.221002    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d20ad712-3573-448c-bab1-c03724bd085b-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.221025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.221014    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d20ad712-3573-448c-bab1-c03724bd085b-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.221025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.221026    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rvw2\" (UniqueName: \"kubernetes.io/projected/d20ad712-3573-448c-bab1-c03724bd085b-kube-api-access-8rvw2\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.310819 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.310796    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:42:07.422990 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.422953    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-kserve-provision-location\") pod \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") "
Jul 28 17:42:07.422990 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.422994    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tls-certs\") pod \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") "
Jul 28 17:42:07.423283 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.423035    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tmp-dir\") pod \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\" (UID: \"dd1494cd-02cd-4cf2-80ac-5ba00490a923\") "
Jul 28 17:42:07.423574 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.423548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dd1494cd-02cd-4cf2-80ac-5ba00490a923" (UID: "dd1494cd-02cd-4cf2-80ac-5ba00490a923"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.425154 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.425130    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd1494cd-02cd-4cf2-80ac-5ba00490a923" (UID: "dd1494cd-02cd-4cf2-80ac-5ba00490a923"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:42:07.485702 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.485625    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd1494cd-02cd-4cf2-80ac-5ba00490a923" (UID: "dd1494cd-02cd-4cf2-80ac-5ba00490a923"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:42:07.523976 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.523947    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.523976 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.523972    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.523976 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.523981    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd1494cd-02cd-4cf2-80ac-5ba00490a923-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:42:07.629672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.629639    2579 generic.go:356] "Generic (PLEG): container finished" podID="dd1494cd-02cd-4cf2-80ac-5ba00490a923" containerID="57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf" exitCode=0
Jul 28 17:42:07.629864 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.629690    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"
Jul 28 17:42:07.629864 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.629725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd" event={"ID":"dd1494cd-02cd-4cf2-80ac-5ba00490a923","Type":"ContainerDied","Data":"57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf"}
Jul 28 17:42:07.629864 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.629785    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd" event={"ID":"dd1494cd-02cd-4cf2-80ac-5ba00490a923","Type":"ContainerDied","Data":"1593c5b51b6b726cc22c5fa30e99dc699858b432d3f1bba5fb624f37bf99dd5a"}
Jul 28 17:42:07.629864 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.629815    2579 scope.go:117] "RemoveContainer" containerID="57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf"
Jul 28 17:42:07.631645 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.631625    2579 generic.go:356] "Generic (PLEG): container finished" podID="d20ad712-3573-448c-bab1-c03724bd085b" containerID="b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c" exitCode=0
Jul 28 17:42:07.631760 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.631653    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r" event={"ID":"d20ad712-3573-448c-bab1-c03724bd085b","Type":"ContainerDied","Data":"b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c"}
Jul 28 17:42:07.631760 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.631698    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"
Jul 28 17:42:07.631850 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.631701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r" event={"ID":"d20ad712-3573-448c-bab1-c03724bd085b","Type":"ContainerDied","Data":"954eb5c0e02c655c279d9040a751949e5a69f83dc859f18730df1d69ee805daa"}
Jul 28 17:42:07.639254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.639205    2579 scope.go:117] "RemoveContainer" containerID="62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c"
Jul 28 17:42:07.649036 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.649005    2579 scope.go:117] "RemoveContainer" containerID="57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf"
Jul 28 17:42:07.649362 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:42:07.649338    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf\": container with ID starting with 57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf not found: ID does not exist" containerID="57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf"
Jul 28 17:42:07.649448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.649369    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf"} err="failed to get container status \"57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf\": rpc error: code = NotFound desc = could not find container \"57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf\": container with ID starting with 57630b5516ab8f036d23ebfe44576b41e4165d815abbe17dceb2e710393d7fbf not found: ID does not exist"
Jul 28 17:42:07.649448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.649388    2579 scope.go:117] "RemoveContainer" containerID="62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c"
Jul 28 17:42:07.649675 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:42:07.649657    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c\": container with ID starting with 62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c not found: ID does not exist" containerID="62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c"
Jul 28 17:42:07.649732 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.649681    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c"} err="failed to get container status \"62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c\": rpc error: code = NotFound desc = could not find container \"62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c\": container with ID starting with 62997dee54eecaa4bf61da6c7d02f2bcc2049cd6e079774a9e225395235d843c not found: ID does not exist"
Jul 28 17:42:07.649732 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.649698    2579 scope.go:117] "RemoveContainer" containerID="b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c"
Jul 28 17:42:07.657627 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.657602    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"]
Jul 28 17:42:07.658447 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.658427    2579 scope.go:117] "RemoveContainer" containerID="e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e"
Jul 28 17:42:07.662886 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.662856    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-tokenizer-5b6c66d958-v2zkd"]
Jul 28 17:42:07.667134 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.666919    2579 scope.go:117] "RemoveContainer" containerID="b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c"
Jul 28 17:42:07.667226 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:42:07.667207    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c\": container with ID starting with b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c not found: ID does not exist" containerID="b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c"
Jul 28 17:42:07.667268 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.667234    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c"} err="failed to get container status \"b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c\": rpc error: code = NotFound desc = could not find container \"b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c\": container with ID starting with b59fe1bcc4b3b7e8ec40b11bf8aeb58b87e85d2d12972716db897e1d6022154c not found: ID does not exist"
Jul 28 17:42:07.667268 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.667252    2579 scope.go:117] "RemoveContainer" containerID="e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e"
Jul 28 17:42:07.667478 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:42:07.667462    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e\": container with ID starting with e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e not found: ID does not exist" containerID="e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e"
Jul 28 17:42:07.667521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.667480    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e"} err="failed to get container status \"e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e\": rpc error: code = NotFound desc = could not find container \"e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e\": container with ID starting with e127203fba82452736dc79ca5ddc715247b34c09a76329503857a4e40760a98e not found: ID does not exist"
Jul 28 17:42:07.673525 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.673505    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"]
Jul 28 17:42:07.677927 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:07.677907    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-5db69d84d9-t9f4r"]
Jul 28 17:42:08.320072 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:08.320038    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20ad712-3573-448c-bab1-c03724bd085b" path="/var/lib/kubelet/pods/d20ad712-3573-448c-bab1-c03724bd085b/volumes"
Jul 28 17:42:08.320509 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:08.320471    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd1494cd-02cd-4cf2-80ac-5ba00490a923" path="/var/lib/kubelet/pods/dd1494cd-02cd-4cf2-80ac-5ba00490a923/volumes"
Jul 28 17:42:36.739740 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.739707    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"]
Jul 28 17:42:36.740364 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.740349    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20ad712-3573-448c-bab1-c03724bd085b" containerName="main"
Jul 28 17:42:36.740439 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.740368    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20ad712-3573-448c-bab1-c03724bd085b" containerName="main"
Jul 28 17:42:36.740439 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.740382    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20ad712-3573-448c-bab1-c03724bd085b" containerName="storage-initializer"
Jul 28 17:42:36.740439 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.740395    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20ad712-3573-448c-bab1-c03724bd085b" containerName="storage-initializer"
Jul 28 17:42:36.740439 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.740405    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1494cd-02cd-4cf2-80ac-5ba00490a923" containerName="main"
Jul 28 17:42:36.740439 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.740414    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1494cd-02cd-4cf2-80ac-5ba00490a923" containerName="main"
Jul 28 17:42:36.740439 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.740427    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd1494cd-02cd-4cf2-80ac-5ba00490a923" containerName="storage-initializer"
Jul 28 17:42:36.740439 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.740435    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd1494cd-02cd-4cf2-80ac-5ba00490a923" containerName="storage-initializer"
Jul 28 17:42:36.740792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.740542    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd1494cd-02cd-4cf2-80ac-5ba00490a923" containerName="main"
Jul 28 17:42:36.740792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.740554    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20ad712-3573-448c-bab1-c03724bd085b" containerName="main"
Jul 28 17:42:36.743162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.743142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.745830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.745810    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 28 17:42:36.745830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.745824    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 28 17:42:36.747118 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.747100    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-5rwls\""
Jul 28 17:42:36.747281 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.747262    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 28 17:42:36.747281 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.747277    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 28 17:42:36.755619 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.755596    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"]
Jul 28 17:42:36.882797 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.882758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.882986 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.882810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.882986 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.882898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.882986 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.882985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.883178 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.883012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.883178 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.883029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28d3883-2863-46f2-8922-166dc2b14868-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.883178 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.883047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdkjs\" (UniqueName: \"kubernetes.io/projected/c28d3883-2863-46f2-8922-166dc2b14868-kube-api-access-qdkjs\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.983755 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.983722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.983925 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.983761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.983925 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.983779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28d3883-2863-46f2-8922-166dc2b14868-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.983925 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.983797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdkjs\" (UniqueName: \"kubernetes.io/projected/c28d3883-2863-46f2-8922-166dc2b14868-kube-api-access-qdkjs\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.984122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.983921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.984122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.983971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.984122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.984017    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.984296 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.984273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.984344 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.984306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-model-cache\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.984344 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.984331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.984411 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.984390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-home\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.986048 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.986024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-dshm\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.986359 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.986339    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28d3883-2863-46f2-8922-166dc2b14868-tls-certs\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:36.992332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:36.992279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdkjs\" (UniqueName: \"kubernetes.io/projected/c28d3883-2863-46f2-8922-166dc2b14868-kube-api-access-qdkjs\") pod \"tokenizer-migration-path-test-kserve-6666787596-2t499\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:37.053321 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.053280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:37.089563 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.089517    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"]
Jul 28 17:42:37.093253 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.093227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.102864 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.102833    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"]
Jul 28 17:42:37.185834 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.185806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.185993 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.185851    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.185993 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.185917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.204621 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.204599    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"]
Jul 28 17:42:37.206890 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:42:37.206864    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28d3883_2863_46f2_8922_166dc2b14868.slice/crio-8f69432cf627391a67ce098853d88765cc4b23f85608b6c04f168f40ea2a31ec WatchSource:0}: Error finding container 8f69432cf627391a67ce098853d88765cc4b23f85608b6c04f168f40ea2a31ec: Status 404 returned error can't find the container with id 8f69432cf627391a67ce098853d88765cc4b23f85608b6c04f168f40ea2a31ec
Jul 28 17:42:37.286623 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.286591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.286807 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.286634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.286807 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.286656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.286965 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.286941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-kserve-provision-location\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.287034 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.286974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tmp-dir\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.288924 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.288899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tls-certs\") pod \"tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.406865 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.406833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:37.538909 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.538886    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"]
Jul 28 17:42:37.540747 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:42:37.540721    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749e121d_6ea4_490b_b48b_fcdacdfa1aed.slice/crio-cc61bac406c9a80fd18ac4c370fe19abe4db01cf63559e3e7904c0a55924c9ca WatchSource:0}: Error finding container cc61bac406c9a80fd18ac4c370fe19abe4db01cf63559e3e7904c0a55924c9ca: Status 404 returned error can't find the container with id cc61bac406c9a80fd18ac4c370fe19abe4db01cf63559e3e7904c0a55924c9ca
Jul 28 17:42:37.776150 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.776035    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499" event={"ID":"c28d3883-2863-46f2-8922-166dc2b14868","Type":"ContainerStarted","Data":"921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3"}
Jul 28 17:42:37.776150 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.776143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499" event={"ID":"c28d3883-2863-46f2-8922-166dc2b14868","Type":"ContainerStarted","Data":"8f69432cf627391a67ce098853d88765cc4b23f85608b6c04f168f40ea2a31ec"}
Jul 28 17:42:37.777676 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.777651    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4" event={"ID":"749e121d-6ea4-490b-b48b-fcdacdfa1aed","Type":"ContainerStarted","Data":"df0989eb519c0f5a3ee4f1722d488e669d11b9c15ab6c3d8f270fb847ea8c6a9"}
Jul 28 17:42:37.777825 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:37.777697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4" event={"ID":"749e121d-6ea4-490b-b48b-fcdacdfa1aed","Type":"ContainerStarted","Data":"cc61bac406c9a80fd18ac4c370fe19abe4db01cf63559e3e7904c0a55924c9ca"}
Jul 28 17:42:48.838271 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:48.838234    2579 generic.go:356] "Generic (PLEG): container finished" podID="749e121d-6ea4-490b-b48b-fcdacdfa1aed" containerID="df0989eb519c0f5a3ee4f1722d488e669d11b9c15ab6c3d8f270fb847ea8c6a9" exitCode=0
Jul 28 17:42:48.838654 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:48.838298    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4" event={"ID":"749e121d-6ea4-490b-b48b-fcdacdfa1aed","Type":"ContainerDied","Data":"df0989eb519c0f5a3ee4f1722d488e669d11b9c15ab6c3d8f270fb847ea8c6a9"}
Jul 28 17:42:49.847404 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:49.847344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4" event={"ID":"749e121d-6ea4-490b-b48b-fcdacdfa1aed","Type":"ContainerStarted","Data":"48b88a129b6a011e0f1219a221fc6203518bfedf9535e1f589073d95fe686e17"}
Jul 28 17:42:49.848112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:49.847613    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:42:49.867879 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:49.867829    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4" podStartSLOduration=2.616386244 podStartE2EDuration="12.867814062s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.251427818s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:42:49.867161303 +0000 UTC m=+1958.096360422" watchObservedRunningTime="2026-07-28 17:42:49.867814062 +0000 UTC m=+1958.097013158"
Jul 28 17:42:53.869936 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:53.869899    2579 generic.go:356] "Generic (PLEG): container finished" podID="c28d3883-2863-46f2-8922-166dc2b14868" containerID="921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3" exitCode=0
Jul 28 17:42:53.870520 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:53.869976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499" event={"ID":"c28d3883-2863-46f2-8922-166dc2b14868","Type":"ContainerDied","Data":"921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3"}
Jul 28 17:42:54.876971 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:54.876937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499" event={"ID":"c28d3883-2863-46f2-8922-166dc2b14868","Type":"ContainerStarted","Data":"cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d"}
Jul 28 17:42:54.905326 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:54.905275    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499" podStartSLOduration=2.589259225 podStartE2EDuration="18.905259525s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.3160003s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:42:54.903257771 +0000 UTC m=+1963.132456867" watchObservedRunningTime="2026-07-28 17:42:54.905259525 +0000 UTC m=+1963.134458620"
Jul 28 17:42:55.054158 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:55.054124    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:57.054379 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:57.054344    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:57.067442 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:57.067416    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:42:57.905566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:42:57.905533    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:43:20.864678 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:20.864592    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:43:22.277216 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:22.277185    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-312caf2d_tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4_749e121d-6ea4-490b-b48b-fcdacdfa1aed/main/0.log"
Jul 28 17:43:22.514504 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:22.514467    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"]
Jul 28 17:43:22.524373 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:43:22.524338    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:43:22.524373 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:43:22.524373 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:43:22.524373 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d" cmd=["/bin/sleep","15"]
Jul 28 17:43:22.524727 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:22.524386    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499" message="" err=<
Jul 28 17:43:22.524727 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:43:22.524727 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:43:22.524727 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:43:22.524727 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:43:22.524398    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:43:22.524727 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:43:22.524727 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:43:22.524727 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499" podUID="c28d3883-2863-46f2-8922-166dc2b14868" containerName="main" containerID="cri-o://cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d"
Jul 28 17:43:22.524727 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:22.524442    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499" podUID="c28d3883-2863-46f2-8922-166dc2b14868" containerName="main" containerID="cri-o://cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d" gracePeriod=60
Jul 28 17:43:22.530402 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:22.530342    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"]
Jul 28 17:43:22.530569 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:22.530535    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4" podUID="749e121d-6ea4-490b-b48b-fcdacdfa1aed" containerName="main" containerID="cri-o://48b88a129b6a011e0f1219a221fc6203518bfedf9535e1f589073d95fe686e17" gracePeriod=30
Jul 28 17:43:22.869604 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:22.869577    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:43:23.009216 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009191    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-home\") pod \"c28d3883-2863-46f2-8922-166dc2b14868\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") "
Jul 28 17:43:23.009368 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009230    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-dshm\") pod \"c28d3883-2863-46f2-8922-166dc2b14868\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") "
Jul 28 17:43:23.009368 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009260    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28d3883-2863-46f2-8922-166dc2b14868-tls-certs\") pod \"c28d3883-2863-46f2-8922-166dc2b14868\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") "
Jul 28 17:43:23.009368 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdkjs\" (UniqueName: \"kubernetes.io/projected/c28d3883-2863-46f2-8922-166dc2b14868-kube-api-access-qdkjs\") pod \"c28d3883-2863-46f2-8922-166dc2b14868\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") "
Jul 28 17:43:23.009539 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-tmp-dir\") pod \"c28d3883-2863-46f2-8922-166dc2b14868\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") "
Jul 28 17:43:23.009539 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009407    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-kserve-provision-location\") pod \"c28d3883-2863-46f2-8922-166dc2b14868\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") "
Jul 28 17:43:23.009539 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009492    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-model-cache\") pod \"c28d3883-2863-46f2-8922-166dc2b14868\" (UID: \"c28d3883-2863-46f2-8922-166dc2b14868\") "
Jul 28 17:43:23.009681 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009533    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-home" (OuterVolumeSpecName: "home") pod "c28d3883-2863-46f2-8922-166dc2b14868" (UID: "c28d3883-2863-46f2-8922-166dc2b14868"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.009786 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009722    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c28d3883-2863-46f2-8922-166dc2b14868" (UID: "c28d3883-2863-46f2-8922-166dc2b14868"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.009786 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009755    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.009938 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.009910    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-model-cache" (OuterVolumeSpecName: "model-cache") pod "c28d3883-2863-46f2-8922-166dc2b14868" (UID: "c28d3883-2863-46f2-8922-166dc2b14868"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.013296 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.013258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c28d3883-2863-46f2-8922-166dc2b14868-kube-api-access-qdkjs" (OuterVolumeSpecName: "kube-api-access-qdkjs") pod "c28d3883-2863-46f2-8922-166dc2b14868" (UID: "c28d3883-2863-46f2-8922-166dc2b14868"). InnerVolumeSpecName "kube-api-access-qdkjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:23.013296 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.013272    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c28d3883-2863-46f2-8922-166dc2b14868-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c28d3883-2863-46f2-8922-166dc2b14868" (UID: "c28d3883-2863-46f2-8922-166dc2b14868"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:23.013421 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.013296    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-dshm" (OuterVolumeSpecName: "dshm") pod "c28d3883-2863-46f2-8922-166dc2b14868" (UID: "c28d3883-2863-46f2-8922-166dc2b14868"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.024492 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.024465    2579 generic.go:356] "Generic (PLEG): container finished" podID="749e121d-6ea4-490b-b48b-fcdacdfa1aed" containerID="48b88a129b6a011e0f1219a221fc6203518bfedf9535e1f589073d95fe686e17" exitCode=0
Jul 28 17:43:23.024601 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.024496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4" event={"ID":"749e121d-6ea4-490b-b48b-fcdacdfa1aed","Type":"ContainerDied","Data":"48b88a129b6a011e0f1219a221fc6203518bfedf9535e1f589073d95fe686e17"}
Jul 28 17:43:23.026790 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.026479    2579 generic.go:356] "Generic (PLEG): container finished" podID="c28d3883-2863-46f2-8922-166dc2b14868" containerID="cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d" exitCode=0
Jul 28 17:43:23.026790 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.026560    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"
Jul 28 17:43:23.026790 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.026592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499" event={"ID":"c28d3883-2863-46f2-8922-166dc2b14868","Type":"ContainerDied","Data":"cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d"}
Jul 28 17:43:23.026790 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.026677    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499" event={"ID":"c28d3883-2863-46f2-8922-166dc2b14868","Type":"ContainerDied","Data":"8f69432cf627391a67ce098853d88765cc4b23f85608b6c04f168f40ea2a31ec"}
Jul 28 17:43:23.026790 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.026714    2579 scope.go:117] "RemoveContainer" containerID="cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d"
Jul 28 17:43:23.038785 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.038761    2579 scope.go:117] "RemoveContainer" containerID="921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3"
Jul 28 17:43:23.047491 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.047471    2579 scope.go:117] "RemoveContainer" containerID="cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d"
Jul 28 17:43:23.047765 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:43:23.047740    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d\": container with ID starting with cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d not found: ID does not exist" containerID="cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d"
Jul 28 17:43:23.047830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.047778    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d"} err="failed to get container status \"cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d\": rpc error: code = NotFound desc = could not find container \"cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d\": container with ID starting with cd536b68416ccecc64cfc2d41d45d20e3088cde909ec7b8cac51ed70a7d4bf6d not found: ID does not exist"
Jul 28 17:43:23.047830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.047802    2579 scope.go:117] "RemoveContainer" containerID="921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3"
Jul 28 17:43:23.048095 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:43:23.048031    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3\": container with ID starting with 921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3 not found: ID does not exist" containerID="921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3"
Jul 28 17:43:23.048205 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.048104    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3"} err="failed to get container status \"921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3\": rpc error: code = NotFound desc = could not find container \"921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3\": container with ID starting with 921a3f6975ab150c5195661e92e04aa6cd592b52c90cd517016a2a34473163a3 not found: ID does not exist"
Jul 28 17:43:23.082361 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.082337    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c28d3883-2863-46f2-8922-166dc2b14868" (UID: "c28d3883-2863-46f2-8922-166dc2b14868"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.083651 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.083634    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:43:23.110853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.110832    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.110853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.110854    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.110992 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.110864    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.110992 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.110872    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c28d3883-2863-46f2-8922-166dc2b14868-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.110992 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.110880    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdkjs\" (UniqueName: \"kubernetes.io/projected/c28d3883-2863-46f2-8922-166dc2b14868-kube-api-access-qdkjs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.110992 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.110908    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c28d3883-2863-46f2-8922-166dc2b14868-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.211584 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.211558    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-kserve-provision-location\") pod \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") "
Jul 28 17:43:23.211741 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.211593    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tmp-dir\") pod \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") "
Jul 28 17:43:23.211741 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.211637    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tls-certs\") pod \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\" (UID: \"749e121d-6ea4-490b-b48b-fcdacdfa1aed\") "
Jul 28 17:43:23.211951 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.211919    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "749e121d-6ea4-490b-b48b-fcdacdfa1aed" (UID: "749e121d-6ea4-490b-b48b-fcdacdfa1aed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.213608 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.213590    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "749e121d-6ea4-490b-b48b-fcdacdfa1aed" (UID: "749e121d-6ea4-490b-b48b-fcdacdfa1aed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:23.267681 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.267645    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "749e121d-6ea4-490b-b48b-fcdacdfa1aed" (UID: "749e121d-6ea4-490b-b48b-fcdacdfa1aed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:23.312332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.312259    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.312332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.312282    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.312332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.312292    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/749e121d-6ea4-490b-b48b-fcdacdfa1aed-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:43:23.350465 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.350441    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"]
Jul 28 17:43:23.356567 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:23.356546    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-6666787596-2t499"]
Jul 28 17:43:24.034173 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:24.034095    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"
Jul 28 17:43:24.034333 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:24.034096    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4" event={"ID":"749e121d-6ea4-490b-b48b-fcdacdfa1aed","Type":"ContainerDied","Data":"cc61bac406c9a80fd18ac4c370fe19abe4db01cf63559e3e7904c0a55924c9ca"}
Jul 28 17:43:24.034333 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:24.034224    2579 scope.go:117] "RemoveContainer" containerID="48b88a129b6a011e0f1219a221fc6203518bfedf9535e1f589073d95fe686e17"
Jul 28 17:43:24.045312 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:24.045292    2579 scope.go:117] "RemoveContainer" containerID="df0989eb519c0f5a3ee4f1722d488e669d11b9c15ab6c3d8f270fb847ea8c6a9"
Jul 28 17:43:24.068131 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:24.068105    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"]
Jul 28 17:43:24.072665 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:24.072645    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-tokenizer-859d58fb96-qfnn4"]
Jul 28 17:43:24.320783 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:24.320750    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="749e121d-6ea4-490b-b48b-fcdacdfa1aed" path="/var/lib/kubelet/pods/749e121d-6ea4-490b-b48b-fcdacdfa1aed/volumes"
Jul 28 17:43:24.321304 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:43:24.321175    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c28d3883-2863-46f2-8922-166dc2b14868" path="/var/lib/kubelet/pods/c28d3883-2863-46f2-8922-166dc2b14868/volumes"
Jul 28 17:44:04.053748 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.053714    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"]
Jul 28 17:44:04.054261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.054066    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="749e121d-6ea4-490b-b48b-fcdacdfa1aed" containerName="main"
Jul 28 17:44:04.054261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.054090    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="749e121d-6ea4-490b-b48b-fcdacdfa1aed" containerName="main"
Jul 28 17:44:04.054261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.054116    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="749e121d-6ea4-490b-b48b-fcdacdfa1aed" containerName="storage-initializer"
Jul 28 17:44:04.054261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.054126    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="749e121d-6ea4-490b-b48b-fcdacdfa1aed" containerName="storage-initializer"
Jul 28 17:44:04.054261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.054136    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c28d3883-2863-46f2-8922-166dc2b14868" containerName="main"
Jul 28 17:44:04.054261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.054144    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c28d3883-2863-46f2-8922-166dc2b14868" containerName="main"
Jul 28 17:44:04.054261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.054164    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c28d3883-2863-46f2-8922-166dc2b14868" containerName="storage-initializer"
Jul 28 17:44:04.054261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.054169    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c28d3883-2863-46f2-8922-166dc2b14868" containerName="storage-initializer"
Jul 28 17:44:04.054261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.054227    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c28d3883-2863-46f2-8922-166dc2b14868" containerName="main"
Jul 28 17:44:04.054261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.054237    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="749e121d-6ea4-490b-b48b-fcdacdfa1aed" containerName="main"
Jul 28 17:44:04.056350 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.056334    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.060349 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.060323    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 28 17:44:04.060349 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.060343    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 28 17:44:04.060558 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.060389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-8b4z2\""
Jul 28 17:44:04.060558 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.060391    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 28 17:44:04.069130 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.069106    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"]
Jul 28 17:44:04.164811 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.164776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.164971 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.164835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.164971 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.164888    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8674948e-43dd-4a90-bdbd-ee4db91eb501-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.164971 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.164953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb2bj\" (UniqueName: \"kubernetes.io/projected/8674948e-43dd-4a90-bdbd-ee4db91eb501-kube-api-access-kb2bj\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.165101 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.165004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.165101 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.165037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.265653 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.265614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb2bj\" (UniqueName: \"kubernetes.io/projected/8674948e-43dd-4a90-bdbd-ee4db91eb501-kube-api-access-kb2bj\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.265653 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.265659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.265681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.265704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.265749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.265895 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.265769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8674948e-43dd-4a90-bdbd-ee4db91eb501-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.266136 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.266057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-model-cache\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.266181 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.266133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-home\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.266242 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.266225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-tmp-dir\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.267983 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.267964    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-dshm\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.268149 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.268133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8674948e-43dd-4a90-bdbd-ee4db91eb501-tls-certs\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.276978 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.276952    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb2bj\" (UniqueName: \"kubernetes.io/projected/8674948e-43dd-4a90-bdbd-ee4db91eb501-kube-api-access-kb2bj\") pod \"llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.297181 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.297156    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"]
Jul 28 17:44:04.299862 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.299845    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:04.303106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.303068    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-zvrng\""
Jul 28 17:44:04.312568 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.312540    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"]
Jul 28 17:44:04.366881 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.366853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:04.367074 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.367054    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55561671-f61b-451b-a6b6-e8a29d43d59d-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh\" (UID: \"55561671-f61b-451b-a6b6-e8a29d43d59d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:04.367216 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.367158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ws9v\" (UniqueName: \"kubernetes.io/projected/55561671-f61b-451b-a6b6-e8a29d43d59d-kube-api-access-7ws9v\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh\" (UID: \"55561671-f61b-451b-a6b6-e8a29d43d59d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:04.467723 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.467683    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55561671-f61b-451b-a6b6-e8a29d43d59d-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh\" (UID: \"55561671-f61b-451b-a6b6-e8a29d43d59d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:04.467890 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.467742    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ws9v\" (UniqueName: \"kubernetes.io/projected/55561671-f61b-451b-a6b6-e8a29d43d59d-kube-api-access-7ws9v\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh\" (UID: \"55561671-f61b-451b-a6b6-e8a29d43d59d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:04.470287 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.470260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55561671-f61b-451b-a6b6-e8a29d43d59d-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh\" (UID: \"55561671-f61b-451b-a6b6-e8a29d43d59d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:04.477430 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.477402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ws9v\" (UniqueName: \"kubernetes.io/projected/55561671-f61b-451b-a6b6-e8a29d43d59d-kube-api-access-7ws9v\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh\" (UID: \"55561671-f61b-451b-a6b6-e8a29d43d59d\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:04.499574 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.499510    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"]
Jul 28 17:44:04.501857 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:44:04.501832    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8674948e_43dd_4a90_bdbd_ee4db91eb501.slice/crio-8fbe83a6191035b074b8aecca013e86f445773d568bf23025009dcc9cbecfb7c WatchSource:0}: Error finding container 8fbe83a6191035b074b8aecca013e86f445773d568bf23025009dcc9cbecfb7c: Status 404 returned error can't find the container with id 8fbe83a6191035b074b8aecca013e86f445773d568bf23025009dcc9cbecfb7c
Jul 28 17:44:04.609955 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.609926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:04.743321 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:04.743276    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"]
Jul 28 17:44:04.745060 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:44:04.745030    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55561671_f61b_451b_a6b6_e8a29d43d59d.slice/crio-f17ec4fbdf9b98cb1e726f310a0be974e9c8f0cdc870892d864ca7d700ee7af7 WatchSource:0}: Error finding container f17ec4fbdf9b98cb1e726f310a0be974e9c8f0cdc870892d864ca7d700ee7af7: Status 404 returned error can't find the container with id f17ec4fbdf9b98cb1e726f310a0be974e9c8f0cdc870892d864ca7d700ee7af7
Jul 28 17:44:05.229833 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:05.229793    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj" event={"ID":"8674948e-43dd-4a90-bdbd-ee4db91eb501","Type":"ContainerStarted","Data":"35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475"}
Jul 28 17:44:05.230535 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:05.229851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj" event={"ID":"8674948e-43dd-4a90-bdbd-ee4db91eb501","Type":"ContainerStarted","Data":"8fbe83a6191035b074b8aecca013e86f445773d568bf23025009dcc9cbecfb7c"}
Jul 28 17:44:05.231338 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:05.231305    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh" event={"ID":"55561671-f61b-451b-a6b6-e8a29d43d59d","Type":"ContainerStarted","Data":"6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6"}
Jul 28 17:44:05.231462 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:05.231364    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh" event={"ID":"55561671-f61b-451b-a6b6-e8a29d43d59d","Type":"ContainerStarted","Data":"f17ec4fbdf9b98cb1e726f310a0be974e9c8f0cdc870892d864ca7d700ee7af7"}
Jul 28 17:44:05.231462 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:05.231393    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:05.252374 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:05.252326    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj" podStartSLOduration=1.25231407 podStartE2EDuration="1.25231407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:05.25141062 +0000 UTC m=+2033.480609727" watchObservedRunningTime="2026-07-28 17:44:05.25231407 +0000 UTC m=+2033.481513165"
Jul 28 17:44:05.271377 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:05.271337    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh" podStartSLOduration=1.271320678 podStartE2EDuration="1.271320678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:05.270563057 +0000 UTC m=+2033.499762163" watchObservedRunningTime="2026-07-28 17:44:05.271320678 +0000 UTC m=+2033.500519752"
Jul 28 17:44:05.367772 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:05.367741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:14.367918 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:14.367886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:14.380686 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:14.380660    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:15.289541 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:15.289514    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:36.238647 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:36.238567    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:37.366154 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.366110    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"]
Jul 28 17:44:37.370012 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.369982    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"]
Jul 28 17:44:37.376104 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:44:37.376050    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:44:37.376104 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:44:37.376104 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:44:37.376104 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475" cmd=["/bin/sleep","15"]
Jul 28 17:44:37.376373 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.376108    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj" message="" err=<
Jul 28 17:44:37.376373 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:44:37.376373 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:44:37.376373 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:44:37.376373 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:44:37.376120    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:44:37.376373 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:44:37.376373 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:44:37.376373 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj" podUID="8674948e-43dd-4a90-bdbd-ee4db91eb501" containerName="main" containerID="cri-o://35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475"
Jul 28 17:44:37.376373 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.376152    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj" podUID="8674948e-43dd-4a90-bdbd-ee4db91eb501" containerName="main" containerID="cri-o://35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475" gracePeriod=60
Jul 28 17:44:37.727713 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.727691    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:37.769752 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.769722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-home\") pod \"8674948e-43dd-4a90-bdbd-ee4db91eb501\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") "
Jul 28 17:44:37.769902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.769799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-model-cache\") pod \"8674948e-43dd-4a90-bdbd-ee4db91eb501\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") "
Jul 28 17:44:37.769902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.769843    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kb2bj\" (UniqueName: \"kubernetes.io/projected/8674948e-43dd-4a90-bdbd-ee4db91eb501-kube-api-access-kb2bj\") pod \"8674948e-43dd-4a90-bdbd-ee4db91eb501\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") "
Jul 28 17:44:37.769902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.769865    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8674948e-43dd-4a90-bdbd-ee4db91eb501-tls-certs\") pod \"8674948e-43dd-4a90-bdbd-ee4db91eb501\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") "
Jul 28 17:44:37.769902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.769898    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-tmp-dir\") pod \"8674948e-43dd-4a90-bdbd-ee4db91eb501\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") "
Jul 28 17:44:37.770159 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.769927    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-dshm\") pod \"8674948e-43dd-4a90-bdbd-ee4db91eb501\" (UID: \"8674948e-43dd-4a90-bdbd-ee4db91eb501\") "
Jul 28 17:44:37.770159 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.770043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-home" (OuterVolumeSpecName: "home") pod "8674948e-43dd-4a90-bdbd-ee4db91eb501" (UID: "8674948e-43dd-4a90-bdbd-ee4db91eb501"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:44:37.770266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.770150    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-model-cache" (OuterVolumeSpecName: "model-cache") pod "8674948e-43dd-4a90-bdbd-ee4db91eb501" (UID: "8674948e-43dd-4a90-bdbd-ee4db91eb501"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:44:37.770266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.770230    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8674948e-43dd-4a90-bdbd-ee4db91eb501" (UID: "8674948e-43dd-4a90-bdbd-ee4db91eb501"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:44:37.770352 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.770263    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:44:37.770352 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.770281    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:44:37.772128 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.772097    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-dshm" (OuterVolumeSpecName: "dshm") pod "8674948e-43dd-4a90-bdbd-ee4db91eb501" (UID: "8674948e-43dd-4a90-bdbd-ee4db91eb501"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:44:37.772235 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.772196    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8674948e-43dd-4a90-bdbd-ee4db91eb501-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8674948e-43dd-4a90-bdbd-ee4db91eb501" (UID: "8674948e-43dd-4a90-bdbd-ee4db91eb501"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:44:37.772235 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.772198    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8674948e-43dd-4a90-bdbd-ee4db91eb501-kube-api-access-kb2bj" (OuterVolumeSpecName: "kube-api-access-kb2bj") pod "8674948e-43dd-4a90-bdbd-ee4db91eb501" (UID: "8674948e-43dd-4a90-bdbd-ee4db91eb501"). InnerVolumeSpecName "kube-api-access-kb2bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:44:37.871248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.871217    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb2bj\" (UniqueName: \"kubernetes.io/projected/8674948e-43dd-4a90-bdbd-ee4db91eb501-kube-api-access-kb2bj\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:44:37.871248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.871243    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8674948e-43dd-4a90-bdbd-ee4db91eb501-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:44:37.871443 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.871258    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:44:37.871443 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:37.871271    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8674948e-43dd-4a90-bdbd-ee4db91eb501-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:44:38.396112 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:38.396059    2579 generic.go:356] "Generic (PLEG): container finished" podID="8674948e-43dd-4a90-bdbd-ee4db91eb501" containerID="35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475" exitCode=0
Jul 28 17:44:38.396517 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:38.396121    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"
Jul 28 17:44:38.396517 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:38.396143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj" event={"ID":"8674948e-43dd-4a90-bdbd-ee4db91eb501","Type":"ContainerDied","Data":"35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475"}
Jul 28 17:44:38.396517 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:38.396196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj" event={"ID":"8674948e-43dd-4a90-bdbd-ee4db91eb501","Type":"ContainerDied","Data":"8fbe83a6191035b074b8aecca013e86f445773d568bf23025009dcc9cbecfb7c"}
Jul 28 17:44:38.396517 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:38.396227    2579 scope.go:117] "RemoveContainer" containerID="35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475"
Jul 28 17:44:38.404574 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:38.404550    2579 scope.go:117] "RemoveContainer" containerID="35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475"
Jul 28 17:44:38.404832 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:44:38.404808    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475\": container with ID starting with 35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475 not found: ID does not exist" containerID="35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475"
Jul 28 17:44:38.404888 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:38.404841    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475"} err="failed to get container status \"35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475\": rpc error: code = NotFound desc = could not find container \"35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475\": container with ID starting with 35d27f940efd2e0a745a8d882eaab48af87bbb4a7eb9f3804134b8a897a50475 not found: ID does not exist"
Jul 28 17:44:38.417470 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:38.417447    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"]
Jul 28 17:44:38.422212 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:38.422189    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc-router-managed-test-llm-e45d1f79-kserve-df8675f66-78kdj"]
Jul 28 17:44:40.321213 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:40.321170    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8674948e-43dd-4a90-bdbd-ee4db91eb501" path="/var/lib/kubelet/pods/8674948e-43dd-4a90-bdbd-ee4db91eb501/volumes"
Jul 28 17:44:46.237739 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:46.237701    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh" podUID="55561671-f61b-451b-a6b6-e8a29d43d59d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:44:52.389777 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:52.389725    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh" podUID="55561671-f61b-451b-a6b6-e8a29d43d59d" containerName="main" containerID="cri-o://6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6" gracePeriod=45
Jul 28 17:44:52.648579 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:52.648524    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:52.797585 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:52.797556    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55561671-f61b-451b-a6b6-e8a29d43d59d-tls-certs\") pod \"55561671-f61b-451b-a6b6-e8a29d43d59d\" (UID: \"55561671-f61b-451b-a6b6-e8a29d43d59d\") "
Jul 28 17:44:52.797737 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:52.797602    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ws9v\" (UniqueName: \"kubernetes.io/projected/55561671-f61b-451b-a6b6-e8a29d43d59d-kube-api-access-7ws9v\") pod \"55561671-f61b-451b-a6b6-e8a29d43d59d\" (UID: \"55561671-f61b-451b-a6b6-e8a29d43d59d\") "
Jul 28 17:44:52.799714 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:52.799682    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55561671-f61b-451b-a6b6-e8a29d43d59d-kube-api-access-7ws9v" (OuterVolumeSpecName: "kube-api-access-7ws9v") pod "55561671-f61b-451b-a6b6-e8a29d43d59d" (UID: "55561671-f61b-451b-a6b6-e8a29d43d59d"). InnerVolumeSpecName "kube-api-access-7ws9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:44:52.799714 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:52.799693    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55561671-f61b-451b-a6b6-e8a29d43d59d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55561671-f61b-451b-a6b6-e8a29d43d59d" (UID: "55561671-f61b-451b-a6b6-e8a29d43d59d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:44:52.898952 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:52.898887    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55561671-f61b-451b-a6b6-e8a29d43d59d-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:44:52.898952 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:52.898909    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ws9v\" (UniqueName: \"kubernetes.io/projected/55561671-f61b-451b-a6b6-e8a29d43d59d-kube-api-access-7ws9v\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:44:53.471493 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:53.471456    2579 generic.go:356] "Generic (PLEG): container finished" podID="55561671-f61b-451b-a6b6-e8a29d43d59d" containerID="6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6" exitCode=0
Jul 28 17:44:53.471970 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:53.471503    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"
Jul 28 17:44:53.471970 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:53.471539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh" event={"ID":"55561671-f61b-451b-a6b6-e8a29d43d59d","Type":"ContainerDied","Data":"6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6"}
Jul 28 17:44:53.471970 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:53.471593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh" event={"ID":"55561671-f61b-451b-a6b6-e8a29d43d59d","Type":"ContainerDied","Data":"f17ec4fbdf9b98cb1e726f310a0be974e9c8f0cdc870892d864ca7d700ee7af7"}
Jul 28 17:44:53.471970 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:53.471617    2579 scope.go:117] "RemoveContainer" containerID="6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6"
Jul 28 17:44:53.480619 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:53.480603    2579 scope.go:117] "RemoveContainer" containerID="6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6"
Jul 28 17:44:53.480853 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:44:53.480838    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6\": container with ID starting with 6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6 not found: ID does not exist" containerID="6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6"
Jul 28 17:44:53.480894 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:53.480864    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6"} err="failed to get container status \"6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6\": rpc error: code = NotFound desc = could not find container \"6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6\": container with ID starting with 6bc92704b2669aba2630d4a2d7197db3f742e791306fc4339348c6e8cb1fb1d6 not found: ID does not exist"
Jul 28 17:44:53.492877 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:53.492853    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"]
Jul 28 17:44:53.496418 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:53.496397    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelfvwh"]
Jul 28 17:44:54.319795 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:44:54.319756    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55561671-f61b-451b-a6b6-e8a29d43d59d" path="/var/lib/kubelet/pods/55561671-f61b-451b-a6b6-e8a29d43d59d/volumes"
Jul 28 17:45:10.464134 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.464097    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"]
Jul 28 17:45:10.464633 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.464481    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8674948e-43dd-4a90-bdbd-ee4db91eb501" containerName="main"
Jul 28 17:45:10.464633 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.464492    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8674948e-43dd-4a90-bdbd-ee4db91eb501" containerName="main"
Jul 28 17:45:10.464633 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.464502    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55561671-f61b-451b-a6b6-e8a29d43d59d" containerName="main"
Jul 28 17:45:10.464633 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.464507    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="55561671-f61b-451b-a6b6-e8a29d43d59d" containerName="main"
Jul 28 17:45:10.464633 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.464565    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="55561671-f61b-451b-a6b6-e8a29d43d59d" containerName="main"
Jul 28 17:45:10.464633 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.464577    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8674948e-43dd-4a90-bdbd-ee4db91eb501" containerName="main"
Jul 28 17:45:10.467321 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.467297    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.472278 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.472257    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-7tvkc\""
Jul 28 17:45:10.472401 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.472367    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 28 17:45:10.472401 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.472367    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 28 17:45:10.472496 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.472479    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 28 17:45:10.484343 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.484315    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"]
Jul 28 17:45:10.546970 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.546939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmn6\" (UniqueName: \"kubernetes.io/projected/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-kube-api-access-jnmn6\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.547200 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.546976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.547200 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.547052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.547200 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.547108    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.547200 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.547164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.547200 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.547193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.647891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.647858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.647891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.647900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.648199 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.647927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmn6\" (UniqueName: \"kubernetes.io/projected/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-kube-api-access-jnmn6\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.648199 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.647945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.648199 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.648122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.648199 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.648177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.648416 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.648349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.648416 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.648373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.648564 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.648546    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.650139 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.650112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.650473 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.650454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.658044 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.658020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmn6\" (UniqueName: \"kubernetes.io/projected/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-kube-api-access-jnmn6\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.739053 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.738976    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"]
Jul 28 17:45:10.748581 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.748554    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:45:10.751405 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.751382    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-x4l7w\""
Jul 28 17:45:10.759558 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.759532    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"]
Jul 28 17:45:10.777738 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.777702    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:10.849180 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.849139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm\" (UID: \"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:45:10.849391 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.849204    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5xvx\" (UniqueName: \"kubernetes.io/projected/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-kube-api-access-l5xvx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm\" (UID: \"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:45:10.933185 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.933149    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"]
Jul 28 17:45:10.936945 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:45:10.936917    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1c44e4_c5a3_4934_a163_18ccae7c6d49.slice/crio-f28eb68a5c52e774798405d3c742ba9657f930b218575fee43b8b3b1e91f9ce1 WatchSource:0}: Error finding container f28eb68a5c52e774798405d3c742ba9657f930b218575fee43b8b3b1e91f9ce1: Status 404 returned error can't find the container with id f28eb68a5c52e774798405d3c742ba9657f930b218575fee43b8b3b1e91f9ce1
Jul 28 17:45:10.950066 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.950033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5xvx\" (UniqueName: \"kubernetes.io/projected/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-kube-api-access-l5xvx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm\" (UID: \"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:45:10.950221 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.950177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm\" (UID: \"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:45:10.952540 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.952522    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm\" (UID: \"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:45:10.963674 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:10.963645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5xvx\" (UniqueName: \"kubernetes.io/projected/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-kube-api-access-l5xvx\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm\" (UID: \"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:45:11.061227 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:11.061189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:45:11.190559 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:11.190473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"]
Jul 28 17:45:11.193189 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:45:11.193163    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ce4c80_f1ee_4590_b04d_aef5f6e7bc7f.slice/crio-6666564f6e304ae50f86d5755b91fe5c51e4000e52d509c92afc9d70ca828fb7 WatchSource:0}: Error finding container 6666564f6e304ae50f86d5755b91fe5c51e4000e52d509c92afc9d70ca828fb7: Status 404 returned error can't find the container with id 6666564f6e304ae50f86d5755b91fe5c51e4000e52d509c92afc9d70ca828fb7
Jul 28 17:45:11.557442 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:11.557343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm" event={"ID":"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f","Type":"ContainerStarted","Data":"281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60"}
Jul 28 17:45:11.557442 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:11.557409    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm" event={"ID":"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f","Type":"ContainerStarted","Data":"6666564f6e304ae50f86d5755b91fe5c51e4000e52d509c92afc9d70ca828fb7"}
Jul 28 17:45:11.557442 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:11.557440    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:45:11.558901 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:11.558877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2" event={"ID":"0b1c44e4-c5a3-4934-a163-18ccae7c6d49","Type":"ContainerStarted","Data":"2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b"}
Jul 28 17:45:11.558948 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:11.558921    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2" event={"ID":"0b1c44e4-c5a3-4934-a163-18ccae7c6d49","Type":"ContainerStarted","Data":"f28eb68a5c52e774798405d3c742ba9657f930b218575fee43b8b3b1e91f9ce1"}
Jul 28 17:45:11.576401 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:11.576354    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm" podStartSLOduration=1.576340686 podStartE2EDuration="1.576340686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:11.574711107 +0000 UTC m=+2099.803910228" watchObservedRunningTime="2026-07-28 17:45:11.576340686 +0000 UTC m=+2099.805539760"
Jul 28 17:45:11.596507 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:11.596462    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2" podStartSLOduration=1.596447909 podStartE2EDuration="1.596447909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:45:11.594629006 +0000 UTC m=+2099.823828127" watchObservedRunningTime="2026-07-28 17:45:11.596447909 +0000 UTC m=+2099.825647005"
Jul 28 17:45:11.777934 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:11.777897    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:12.358657 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:12.358621    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:45:12.366558 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:12.366531    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:45:20.778845 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:20.778800    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:20.791304 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:20.791276    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:21.625859 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:21.625829    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:45:42.564917 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:45:42.564890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:46:00.438334 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.438299    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"]
Jul 28 17:46:00.441636 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.441620    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.444295 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.444272    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 28 17:46:00.452388 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.452361    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"]
Jul 28 17:46:00.490883 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.490856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.490883 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.490883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d633b0-b606-4f62-9014-1868b8f7912c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.491103 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.490906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.491103 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.490926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blk96\" (UniqueName: \"kubernetes.io/projected/00d633b0-b606-4f62-9014-1868b8f7912c-kube-api-access-blk96\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.491103 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.491045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.491261 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.491127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.592529 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.592493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.592720 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.592539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.592720 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.592588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.592720 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.592610    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d633b0-b606-4f62-9014-1868b8f7912c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.592720 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.592637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.592933 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.592792    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blk96\" (UniqueName: \"kubernetes.io/projected/00d633b0-b606-4f62-9014-1868b8f7912c-kube-api-access-blk96\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.592998 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.592928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.592998 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.592941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.593121 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.593021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.594753 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.594737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.595106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.595070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d633b0-b606-4f62-9014-1868b8f7912c-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.601353 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.601331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blk96\" (UniqueName: \"kubernetes.io/projected/00d633b0-b606-4f62-9014-1868b8f7912c-kube-api-access-blk96\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.753201 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.753118    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:00.795299 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.795265    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"]
Jul 28 17:46:00.800039 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.799923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:00.802839 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.802816    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-t9qr7\""
Jul 28 17:46:00.819973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.819931    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"]
Jul 28 17:46:00.896572 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.896535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67533af9-994f-4d0c-bee6-27815e05e32c-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94\" (UID: \"67533af9-994f-4d0c-bee6-27815e05e32c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:00.896748 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.896694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwrhn\" (UniqueName: \"kubernetes.io/projected/67533af9-994f-4d0c-bee6-27815e05e32c-kube-api-access-hwrhn\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94\" (UID: \"67533af9-994f-4d0c-bee6-27815e05e32c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:00.900147 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.900109    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"]
Jul 28 17:46:00.903353 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:46:00.903318    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d633b0_b606_4f62_9014_1868b8f7912c.slice/crio-00d99aaa06f4ac4ef956897cc42c43d56e764eecf080f818a261c0e97c3e03d2 WatchSource:0}: Error finding container 00d99aaa06f4ac4ef956897cc42c43d56e764eecf080f818a261c0e97c3e03d2: Status 404 returned error can't find the container with id 00d99aaa06f4ac4ef956897cc42c43d56e764eecf080f818a261c0e97c3e03d2
Jul 28 17:46:00.997694 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.997657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwrhn\" (UniqueName: \"kubernetes.io/projected/67533af9-994f-4d0c-bee6-27815e05e32c-kube-api-access-hwrhn\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94\" (UID: \"67533af9-994f-4d0c-bee6-27815e05e32c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:00.997883 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:00.997774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67533af9-994f-4d0c-bee6-27815e05e32c-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94\" (UID: \"67533af9-994f-4d0c-bee6-27815e05e32c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:01.000313 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.000289    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67533af9-994f-4d0c-bee6-27815e05e32c-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94\" (UID: \"67533af9-994f-4d0c-bee6-27815e05e32c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:01.005466 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.005412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwrhn\" (UniqueName: \"kubernetes.io/projected/67533af9-994f-4d0c-bee6-27815e05e32c-kube-api-access-hwrhn\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94\" (UID: \"67533af9-994f-4d0c-bee6-27815e05e32c\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:01.115155 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.115120    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:01.241268 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.241239    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"]
Jul 28 17:46:01.243870 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:46:01.243842    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67533af9_994f_4d0c_bee6_27815e05e32c.slice/crio-bc3cb4cc027937b200ae24f680e6cff26fceae74663affa700a7e12501117826 WatchSource:0}: Error finding container bc3cb4cc027937b200ae24f680e6cff26fceae74663affa700a7e12501117826: Status 404 returned error can't find the container with id bc3cb4cc027937b200ae24f680e6cff26fceae74663affa700a7e12501117826
Jul 28 17:46:01.811715 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.811673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94" event={"ID":"67533af9-994f-4d0c-bee6-27815e05e32c","Type":"ContainerStarted","Data":"fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6"}
Jul 28 17:46:01.812244 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.811840    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:01.812244 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.811872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94" event={"ID":"67533af9-994f-4d0c-bee6-27815e05e32c","Type":"ContainerStarted","Data":"bc3cb4cc027937b200ae24f680e6cff26fceae74663affa700a7e12501117826"}
Jul 28 17:46:01.813321 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.813294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz" event={"ID":"00d633b0-b606-4f62-9014-1868b8f7912c","Type":"ContainerStarted","Data":"6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8"}
Jul 28 17:46:01.813422 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.813348    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz" event={"ID":"00d633b0-b606-4f62-9014-1868b8f7912c","Type":"ContainerStarted","Data":"00d99aaa06f4ac4ef956897cc42c43d56e764eecf080f818a261c0e97c3e03d2"}
Jul 28 17:46:01.829758 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.829716    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94" podStartSLOduration=1.82969807 podStartE2EDuration="1.82969807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:01.828411261 +0000 UTC m=+2150.057610382" watchObservedRunningTime="2026-07-28 17:46:01.82969807 +0000 UTC m=+2150.058897157"
Jul 28 17:46:01.853493 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:01.853443    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz" podStartSLOduration=1.853429297 podStartE2EDuration="1.853429297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:01.851687022 +0000 UTC m=+2150.080886120" watchObservedRunningTime="2026-07-28 17:46:01.853429297 +0000 UTC m=+2150.082628393"
Jul 28 17:46:02.753739 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:02.753708    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:10.754408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:10.754190    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:10.770130 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:10.770101    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:10.869882 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:10.869851    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:32.820154 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:32.820121    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:35.257577 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.257542    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"]
Jul 28 17:46:35.265595 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.265567    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"]
Jul 28 17:46:35.274103 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:46:35.274058    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:46:35.274103 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:46:35.274103 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:46:35.274103 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8" cmd=["/bin/sleep","15"]
Jul 28 17:46:35.274363 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.274118    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz" message="" err=<
Jul 28 17:46:35.274363 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:46:35.274363 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:46:35.274363 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:46:35.274363 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:46:35.274133    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:46:35.274363 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:46:35.274363 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:46:35.274363 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz" podUID="00d633b0-b606-4f62-9014-1868b8f7912c" containerName="main" containerID="cri-o://6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8"
Jul 28 17:46:35.274363 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.274169    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz" podUID="00d633b0-b606-4f62-9014-1868b8f7912c" containerName="main" containerID="cri-o://6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8" gracePeriod=60
Jul 28 17:46:35.632164 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.632139    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:35.811857 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.811778    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-dshm\") pod \"00d633b0-b606-4f62-9014-1868b8f7912c\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") "
Jul 28 17:46:35.811857 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.811823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-home\") pod \"00d633b0-b606-4f62-9014-1868b8f7912c\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") "
Jul 28 17:46:35.812122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.811884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-tmp-dir\") pod \"00d633b0-b606-4f62-9014-1868b8f7912c\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") "
Jul 28 17:46:35.812122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.811912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blk96\" (UniqueName: \"kubernetes.io/projected/00d633b0-b606-4f62-9014-1868b8f7912c-kube-api-access-blk96\") pod \"00d633b0-b606-4f62-9014-1868b8f7912c\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") "
Jul 28 17:46:35.812122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.811936    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d633b0-b606-4f62-9014-1868b8f7912c-tls-certs\") pod \"00d633b0-b606-4f62-9014-1868b8f7912c\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") "
Jul 28 17:46:35.812122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.811958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-model-cache\") pod \"00d633b0-b606-4f62-9014-1868b8f7912c\" (UID: \"00d633b0-b606-4f62-9014-1868b8f7912c\") "
Jul 28 17:46:35.812325 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.812195    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "00d633b0-b606-4f62-9014-1868b8f7912c" (UID: "00d633b0-b606-4f62-9014-1868b8f7912c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:35.812325 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.812210    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-home" (OuterVolumeSpecName: "home") pod "00d633b0-b606-4f62-9014-1868b8f7912c" (UID: "00d633b0-b606-4f62-9014-1868b8f7912c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:35.812401 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.812314    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-model-cache" (OuterVolumeSpecName: "model-cache") pod "00d633b0-b606-4f62-9014-1868b8f7912c" (UID: "00d633b0-b606-4f62-9014-1868b8f7912c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:35.814127 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.814099    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-dshm" (OuterVolumeSpecName: "dshm") pod "00d633b0-b606-4f62-9014-1868b8f7912c" (UID: "00d633b0-b606-4f62-9014-1868b8f7912c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:35.814127 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.814115    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d633b0-b606-4f62-9014-1868b8f7912c-kube-api-access-blk96" (OuterVolumeSpecName: "kube-api-access-blk96") pod "00d633b0-b606-4f62-9014-1868b8f7912c" (UID: "00d633b0-b606-4f62-9014-1868b8f7912c"). InnerVolumeSpecName "kube-api-access-blk96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:35.814127 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.814100    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d633b0-b606-4f62-9014-1868b8f7912c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "00d633b0-b606-4f62-9014-1868b8f7912c" (UID: "00d633b0-b606-4f62-9014-1868b8f7912c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:35.913453 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.913422    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:35.913453 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.913450    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:35.913645 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.913458    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:35.913645 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.913479    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00d633b0-b606-4f62-9014-1868b8f7912c-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:35.913645 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.913487    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blk96\" (UniqueName: \"kubernetes.io/projected/00d633b0-b606-4f62-9014-1868b8f7912c-kube-api-access-blk96\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:35.913645 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.913495    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00d633b0-b606-4f62-9014-1868b8f7912c-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:35.977021 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.976986    2579 generic.go:356] "Generic (PLEG): container finished" podID="00d633b0-b606-4f62-9014-1868b8f7912c" containerID="6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8" exitCode=0
Jul 28 17:46:35.977180 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.977034    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"
Jul 28 17:46:35.977180 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.977051    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz" event={"ID":"00d633b0-b606-4f62-9014-1868b8f7912c","Type":"ContainerDied","Data":"6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8"}
Jul 28 17:46:35.977180 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.977115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz" event={"ID":"00d633b0-b606-4f62-9014-1868b8f7912c","Type":"ContainerDied","Data":"00d99aaa06f4ac4ef956897cc42c43d56e764eecf080f818a261c0e97c3e03d2"}
Jul 28 17:46:35.977180 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.977149    2579 scope.go:117] "RemoveContainer" containerID="6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8"
Jul 28 17:46:35.986797 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.986779    2579 scope.go:117] "RemoveContainer" containerID="6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8"
Jul 28 17:46:35.987050 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:46:35.987032    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8\": container with ID starting with 6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8 not found: ID does not exist" containerID="6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8"
Jul 28 17:46:35.987118 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:35.987060    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8"} err="failed to get container status \"6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8\": rpc error: code = NotFound desc = could not find container \"6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8\": container with ID starting with 6c7d8cac6b0734ed8f6ab298abcb4bd231893392e5b36e8674f6d3b233ec2fd8 not found: ID does not exist"
Jul 28 17:46:36.000618 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:36.000597    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"]
Jul 28 17:46:36.005122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:36.005101    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995gltpz"]
Jul 28 17:46:36.320660 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:36.320625    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d633b0-b606-4f62-9014-1868b8f7912c" path="/var/lib/kubelet/pods/00d633b0-b606-4f62-9014-1868b8f7912c/volumes"
Jul 28 17:46:42.820441 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:42.820406    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94" podUID="67533af9-994f-4d0c-bee6-27815e05e32c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:46:50.286319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:50.286268    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94" podUID="67533af9-994f-4d0c-bee6-27815e05e32c" containerName="main" containerID="cri-o://fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6" gracePeriod=45
Jul 28 17:46:50.534907 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:50.534884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:50.634856 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:50.634830    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwrhn\" (UniqueName: \"kubernetes.io/projected/67533af9-994f-4d0c-bee6-27815e05e32c-kube-api-access-hwrhn\") pod \"67533af9-994f-4d0c-bee6-27815e05e32c\" (UID: \"67533af9-994f-4d0c-bee6-27815e05e32c\") "
Jul 28 17:46:50.635030 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:50.634920    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67533af9-994f-4d0c-bee6-27815e05e32c-tls-certs\") pod \"67533af9-994f-4d0c-bee6-27815e05e32c\" (UID: \"67533af9-994f-4d0c-bee6-27815e05e32c\") "
Jul 28 17:46:50.636951 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:50.636923    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67533af9-994f-4d0c-bee6-27815e05e32c-kube-api-access-hwrhn" (OuterVolumeSpecName: "kube-api-access-hwrhn") pod "67533af9-994f-4d0c-bee6-27815e05e32c" (UID: "67533af9-994f-4d0c-bee6-27815e05e32c"). InnerVolumeSpecName "kube-api-access-hwrhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:50.637053 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:50.636932    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67533af9-994f-4d0c-bee6-27815e05e32c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67533af9-994f-4d0c-bee6-27815e05e32c" (UID: "67533af9-994f-4d0c-bee6-27815e05e32c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:50.735765 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:50.735737    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwrhn\" (UniqueName: \"kubernetes.io/projected/67533af9-994f-4d0c-bee6-27815e05e32c-kube-api-access-hwrhn\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:50.735765 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:50.735760    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67533af9-994f-4d0c-bee6-27815e05e32c-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:51.054605 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:51.054518    2579 generic.go:356] "Generic (PLEG): container finished" podID="67533af9-994f-4d0c-bee6-27815e05e32c" containerID="fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6" exitCode=0
Jul 28 17:46:51.054605 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:51.054560    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"
Jul 28 17:46:51.054797 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:51.054602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94" event={"ID":"67533af9-994f-4d0c-bee6-27815e05e32c","Type":"ContainerDied","Data":"fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6"}
Jul 28 17:46:51.054797 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:51.054655    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94" event={"ID":"67533af9-994f-4d0c-bee6-27815e05e32c","Type":"ContainerDied","Data":"bc3cb4cc027937b200ae24f680e6cff26fceae74663affa700a7e12501117826"}
Jul 28 17:46:51.054797 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:51.054678    2579 scope.go:117] "RemoveContainer" containerID="fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6"
Jul 28 17:46:51.064474 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:51.064453    2579 scope.go:117] "RemoveContainer" containerID="fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6"
Jul 28 17:46:51.064739 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:46:51.064718    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6\": container with ID starting with fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6 not found: ID does not exist" containerID="fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6"
Jul 28 17:46:51.064782 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:51.064747    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6"} err="failed to get container status \"fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6\": rpc error: code = NotFound desc = could not find container \"fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6\": container with ID starting with fd3197007bd6f51d0b7bdc78825cfb320d8c1ec5e2771a0fc26fc4eab06fc9a6 not found: ID does not exist"
Jul 28 17:46:51.076981 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:51.076959    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"]
Jul 28 17:46:51.081445 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:51.081425    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheh6j94"]
Jul 28 17:46:52.320391 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:52.320355    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67533af9-994f-4d0c-bee6-27815e05e32c" path="/var/lib/kubelet/pods/67533af9-994f-4d0c-bee6-27815e05e32c/volumes"
Jul 28 17:46:55.275392 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.275361    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"]
Jul 28 17:46:55.280800 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.280770    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"]
Jul 28 17:46:55.284983 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:46:55.284929    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:46:55.284983 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:46:55.284983 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:46:55.284983 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b" cmd=["/bin/sleep","15"]
Jul 28 17:46:55.285439 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.284982    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2" message="" err=<
Jul 28 17:46:55.285439 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:46:55.285439 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:46:55.285439 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:46:55.285439 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:46:55.285002    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:46:55.285439 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:46:55.285439 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:46:55.285439 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2" podUID="0b1c44e4-c5a3-4934-a163-18ccae7c6d49" containerName="main" containerID="cri-o://2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b"
Jul 28 17:46:55.285439 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.285048    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2" podUID="0b1c44e4-c5a3-4934-a163-18ccae7c6d49" containerName="main" containerID="cri-o://2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b" gracePeriod=60
Jul 28 17:46:55.644441 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.644417    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:46:55.679532 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.679508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-home\") pod \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") "
Jul 28 17:46:55.679652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.679578    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-model-cache\") pod \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") "
Jul 28 17:46:55.679652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.679626    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnmn6\" (UniqueName: \"kubernetes.io/projected/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-kube-api-access-jnmn6\") pod \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") "
Jul 28 17:46:55.679652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.679650    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tmp-dir\") pod \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") "
Jul 28 17:46:55.679790 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.679721    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-dshm\") pod \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") "
Jul 28 17:46:55.679790 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.679763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tls-certs\") pod \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\" (UID: \"0b1c44e4-c5a3-4934-a163-18ccae7c6d49\") "
Jul 28 17:46:55.679893 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.679792    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-home" (OuterVolumeSpecName: "home") pod "0b1c44e4-c5a3-4934-a163-18ccae7c6d49" (UID: "0b1c44e4-c5a3-4934-a163-18ccae7c6d49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:55.679893 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.679870    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b1c44e4-c5a3-4934-a163-18ccae7c6d49" (UID: "0b1c44e4-c5a3-4934-a163-18ccae7c6d49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:55.680043 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.679890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b1c44e4-c5a3-4934-a163-18ccae7c6d49" (UID: "0b1c44e4-c5a3-4934-a163-18ccae7c6d49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:55.680117 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.680055    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:55.680117 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.680070    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:55.680117 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.680113    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:55.681688 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.681667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-dshm" (OuterVolumeSpecName: "dshm") pod "0b1c44e4-c5a3-4934-a163-18ccae7c6d49" (UID: "0b1c44e4-c5a3-4934-a163-18ccae7c6d49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:55.681821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.681803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b1c44e4-c5a3-4934-a163-18ccae7c6d49" (UID: "0b1c44e4-c5a3-4934-a163-18ccae7c6d49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:55.682165 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.682145    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-kube-api-access-jnmn6" (OuterVolumeSpecName: "kube-api-access-jnmn6") pod "0b1c44e4-c5a3-4934-a163-18ccae7c6d49" (UID: "0b1c44e4-c5a3-4934-a163-18ccae7c6d49"). InnerVolumeSpecName "kube-api-access-jnmn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:55.781419 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.781383    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmn6\" (UniqueName: \"kubernetes.io/projected/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-kube-api-access-jnmn6\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:55.781419 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.781416    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:55.781419 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:55.781426    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b1c44e4-c5a3-4934-a163-18ccae7c6d49-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:46:56.081319 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.081285    2579 generic.go:356] "Generic (PLEG): container finished" podID="0b1c44e4-c5a3-4934-a163-18ccae7c6d49" containerID="2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b" exitCode=0
Jul 28 17:46:56.081533 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.081327    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"
Jul 28 17:46:56.081533 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.081374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2" event={"ID":"0b1c44e4-c5a3-4934-a163-18ccae7c6d49","Type":"ContainerDied","Data":"2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b"}
Jul 28 17:46:56.081533 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.081431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2" event={"ID":"0b1c44e4-c5a3-4934-a163-18ccae7c6d49","Type":"ContainerDied","Data":"f28eb68a5c52e774798405d3c742ba9657f930b218575fee43b8b3b1e91f9ce1"}
Jul 28 17:46:56.081533 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.081453    2579 scope.go:117] "RemoveContainer" containerID="2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b"
Jul 28 17:46:56.092279 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.092258    2579 scope.go:117] "RemoveContainer" containerID="2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b"
Jul 28 17:46:56.092558 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:46:56.092541    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b\": container with ID starting with 2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b not found: ID does not exist" containerID="2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b"
Jul 28 17:46:56.092615 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.092567    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b"} err="failed to get container status \"2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b\": rpc error: code = NotFound desc = could not find container \"2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b\": container with ID starting with 2c9fb4e1d830fd9e52fdc6f8ece2f5ee8b0189ce62e1c7a5835f080faee0025b not found: ID does not exist"
Jul 28 17:46:56.104839 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.104810    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"]
Jul 28 17:46:56.107072 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.107051    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2"]
Jul 28 17:46:56.325286 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.325250    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1c44e4-c5a3-4934-a163-18ccae7c6d49" path="/var/lib/kubelet/pods/0b1c44e4-c5a3-4934-a163-18ccae7c6d49/volumes"
Jul 28 17:46:56.616354 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:46:56.616302    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-n66c2" podUID="0b1c44e4-c5a3-4934-a163-18ccae7c6d49" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:47:02.565504 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:02.565467    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm" podUID="b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:47:10.300230 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:10.300151    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm" podUID="b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f" containerName="main" containerID="cri-o://281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60" gracePeriod=45
Jul 28 17:47:10.552693 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:10.552642    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:47:10.609711 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:10.609682    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-tls-certs\") pod \"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f\" (UID: \"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f\") "
Jul 28 17:47:10.609834 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:10.609775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5xvx\" (UniqueName: \"kubernetes.io/projected/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-kube-api-access-l5xvx\") pod \"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f\" (UID: \"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f\") "
Jul 28 17:47:10.611763 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:10.611739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f" (UID: "b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:10.611864 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:10.611811    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-kube-api-access-l5xvx" (OuterVolumeSpecName: "kube-api-access-l5xvx") pod "b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f" (UID: "b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f"). InnerVolumeSpecName "kube-api-access-l5xvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:10.710710 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:10.710676    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:47:10.710710 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:10.710705    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5xvx\" (UniqueName: \"kubernetes.io/projected/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f-kube-api-access-l5xvx\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:47:11.154660 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:11.154629    2579 generic.go:356] "Generic (PLEG): container finished" podID="b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f" containerID="281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60" exitCode=0
Jul 28 17:47:11.154845 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:11.154672    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"
Jul 28 17:47:11.154845 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:11.154722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm" event={"ID":"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f","Type":"ContainerDied","Data":"281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60"}
Jul 28 17:47:11.154845 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:11.154767    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm" event={"ID":"b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f","Type":"ContainerDied","Data":"6666564f6e304ae50f86d5755b91fe5c51e4000e52d509c92afc9d70ca828fb7"}
Jul 28 17:47:11.154845 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:11.154788    2579 scope.go:117] "RemoveContainer" containerID="281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60"
Jul 28 17:47:11.165803 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:11.165737    2579 scope.go:117] "RemoveContainer" containerID="281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60"
Jul 28 17:47:11.166192 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:47:11.166127    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60\": container with ID starting with 281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60 not found: ID does not exist" containerID="281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60"
Jul 28 17:47:11.166192 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:11.166164    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60"} err="failed to get container status \"281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60\": rpc error: code = NotFound desc = could not find container \"281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60\": container with ID starting with 281710bf9aab4ddf9b23f9c47b505f004bb7e105f037a0c52c3b91300f9f6c60 not found: ID does not exist"
Jul 28 17:47:11.176938 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:11.176910    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"]
Jul 28 17:47:11.180745 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:11.180721    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schenfmzm"]
Jul 28 17:47:12.320765 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:12.320731    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f" path="/var/lib/kubelet/pods/b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f/volumes"
Jul 28 17:47:19.951259 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951226    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"]
Jul 28 17:47:19.951664 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951610    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f" containerName="main"
Jul 28 17:47:19.951664 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951624    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f" containerName="main"
Jul 28 17:47:19.951664 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951641    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1c44e4-c5a3-4934-a163-18ccae7c6d49" containerName="main"
Jul 28 17:47:19.951664 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951646    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1c44e4-c5a3-4934-a163-18ccae7c6d49" containerName="main"
Jul 28 17:47:19.951664 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951653    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67533af9-994f-4d0c-bee6-27815e05e32c" containerName="main"
Jul 28 17:47:19.951664 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951658    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="67533af9-994f-4d0c-bee6-27815e05e32c" containerName="main"
Jul 28 17:47:19.951858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951678    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d633b0-b606-4f62-9014-1868b8f7912c" containerName="main"
Jul 28 17:47:19.951858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951684    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d633b0-b606-4f62-9014-1868b8f7912c" containerName="main"
Jul 28 17:47:19.951858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951737    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0ce4c80-f1ee-4590-b04d-aef5f6e7bc7f" containerName="main"
Jul 28 17:47:19.951858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951746    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1c44e4-c5a3-4934-a163-18ccae7c6d49" containerName="main"
Jul 28 17:47:19.951858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951754    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d633b0-b606-4f62-9014-1868b8f7912c" containerName="main"
Jul 28 17:47:19.951858 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.951759    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="67533af9-994f-4d0c-bee6-27815e05e32c" containerName="main"
Jul 28 17:47:19.954634 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.954614    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:19.958860 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.958841    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 28 17:47:19.958973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.958841    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 28 17:47:19.958973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.958870    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 28 17:47:19.958973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.958864    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-s22hh\""
Jul 28 17:47:19.964385 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:19.964362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"]
Jul 28 17:47:20.093396 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.093360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.093396 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.093398    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.093638 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.093430    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85a89a6-39ae-4176-aec9-c78c5334b391-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.093638 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.093494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.093638 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.093550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.093638 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.093580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6cd\" (UniqueName: \"kubernetes.io/projected/b85a89a6-39ae-4176-aec9-c78c5334b391-kube-api-access-7v6cd\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.156644 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.156612    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"]
Jul 28 17:47:20.160073 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.160051    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:47:20.162923 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.162900    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-tpfnb\""
Jul 28 17:47:20.169344 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.169324    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"]
Jul 28 17:47:20.194695 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.194670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.194792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.194711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6cd\" (UniqueName: \"kubernetes.io/projected/b85a89a6-39ae-4176-aec9-c78c5334b391-kube-api-access-7v6cd\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.194843 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.194809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.194843 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.194835    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.194927 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.194868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85a89a6-39ae-4176-aec9-c78c5334b391-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.194927 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.194920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.195088 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.195039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.195221 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.195202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.195276 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.195228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.197229 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.197206    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.197507 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.197485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85a89a6-39ae-4176-aec9-c78c5334b391-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.209184 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.209124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6cd\" (UniqueName: \"kubernetes.io/projected/b85a89a6-39ae-4176-aec9-c78c5334b391-kube-api-access-7v6cd\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.264860 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.264833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:20.295640 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.295603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34cc5f4-a485-4153-bb14-b4404f9eb92b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h\" (UID: \"d34cc5f4-a485-4153-bb14-b4404f9eb92b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:47:20.295773 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.295668    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbqj\" (UniqueName: \"kubernetes.io/projected/d34cc5f4-a485-4153-bb14-b4404f9eb92b-kube-api-access-jmbqj\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h\" (UID: \"d34cc5f4-a485-4153-bb14-b4404f9eb92b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:47:20.396424 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.396391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34cc5f4-a485-4153-bb14-b4404f9eb92b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h\" (UID: \"d34cc5f4-a485-4153-bb14-b4404f9eb92b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:47:20.396594 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.396436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbqj\" (UniqueName: \"kubernetes.io/projected/d34cc5f4-a485-4153-bb14-b4404f9eb92b-kube-api-access-jmbqj\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h\" (UID: \"d34cc5f4-a485-4153-bb14-b4404f9eb92b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:47:20.398965 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.398935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34cc5f4-a485-4153-bb14-b4404f9eb92b-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h\" (UID: \"d34cc5f4-a485-4153-bb14-b4404f9eb92b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:47:20.404285 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.404262    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"]
Jul 28 17:47:20.406507 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.406485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbqj\" (UniqueName: \"kubernetes.io/projected/d34cc5f4-a485-4153-bb14-b4404f9eb92b-kube-api-access-jmbqj\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h\" (UID: \"d34cc5f4-a485-4153-bb14-b4404f9eb92b\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:47:20.407027 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:47:20.407003    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85a89a6_39ae_4176_aec9_c78c5334b391.slice/crio-957fce5a0b9afe5100395cf4a10181e62d458f6b7576caa7416702c6839b666f WatchSource:0}: Error finding container 957fce5a0b9afe5100395cf4a10181e62d458f6b7576caa7416702c6839b666f: Status 404 returned error can't find the container with id 957fce5a0b9afe5100395cf4a10181e62d458f6b7576caa7416702c6839b666f
Jul 28 17:47:20.470957 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.470935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:47:20.608002 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:20.607977    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"]
Jul 28 17:47:20.610188 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:47:20.610164    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34cc5f4_a485_4153_bb14_b4404f9eb92b.slice/crio-1f418c9c0b74827710a4ec89e1e58303cc115881743dd053c222f929755f0fa6 WatchSource:0}: Error finding container 1f418c9c0b74827710a4ec89e1e58303cc115881743dd053c222f929755f0fa6: Status 404 returned error can't find the container with id 1f418c9c0b74827710a4ec89e1e58303cc115881743dd053c222f929755f0fa6
Jul 28 17:47:21.211317 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:21.211249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h" event={"ID":"d34cc5f4-a485-4153-bb14-b4404f9eb92b","Type":"ContainerStarted","Data":"82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050"}
Jul 28 17:47:21.211781 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:21.211338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h" event={"ID":"d34cc5f4-a485-4153-bb14-b4404f9eb92b","Type":"ContainerStarted","Data":"1f418c9c0b74827710a4ec89e1e58303cc115881743dd053c222f929755f0fa6"}
Jul 28 17:47:21.211781 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:21.211375    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:47:21.213023 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:21.212988    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c" event={"ID":"b85a89a6-39ae-4176-aec9-c78c5334b391","Type":"ContainerStarted","Data":"ccc6dc47031f98f6018e703b745d3e053624df588b460dfc40900bb23768d80a"}
Jul 28 17:47:21.213165 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:21.213050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c" event={"ID":"b85a89a6-39ae-4176-aec9-c78c5334b391","Type":"ContainerStarted","Data":"957fce5a0b9afe5100395cf4a10181e62d458f6b7576caa7416702c6839b666f"}
Jul 28 17:47:21.248181 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:21.248117    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h" podStartSLOduration=1.2481025159999999 podStartE2EDuration="1.248102516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:21.246068168 +0000 UTC m=+2229.475267264" watchObservedRunningTime="2026-07-28 17:47:21.248102516 +0000 UTC m=+2229.477301612"
Jul 28 17:47:21.265777 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:21.265736    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:21.266413 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:21.266379    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c" podStartSLOduration=2.26637069 podStartE2EDuration="2.26637069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:21.264747905 +0000 UTC m=+2229.493947002" watchObservedRunningTime="2026-07-28 17:47:21.26637069 +0000 UTC m=+2229.495569785"
Jul 28 17:47:30.265965 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:30.265932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:30.278278 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:30.278246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:31.272025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:31.271992    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:47:52.219121 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:47:52.219089    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:48:06.781592 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:06.781551    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"]
Jul 28 17:48:06.785897 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:06.785876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:48:06.790115 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:06.790072    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-vtz22\""
Jul 28 17:48:06.790248 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:06.790152    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 28 17:48:06.795332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:06.795307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"]
Jul 28 17:48:06.909838 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:06.909803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czpjr\" (UniqueName: \"kubernetes.io/projected/91332062-7be0-45b2-b02b-e4258bb756bd-kube-api-access-czpjr\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm\" (UID: \"91332062-7be0-45b2-b02b-e4258bb756bd\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:48:06.910015 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:06.909847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91332062-7be0-45b2-b02b-e4258bb756bd-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm\" (UID: \"91332062-7be0-45b2-b02b-e4258bb756bd\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:48:07.010501 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:07.010462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czpjr\" (UniqueName: \"kubernetes.io/projected/91332062-7be0-45b2-b02b-e4258bb756bd-kube-api-access-czpjr\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm\" (UID: \"91332062-7be0-45b2-b02b-e4258bb756bd\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:48:07.010667 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:07.010507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91332062-7be0-45b2-b02b-e4258bb756bd-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm\" (UID: \"91332062-7be0-45b2-b02b-e4258bb756bd\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:48:07.012957 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:07.012931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91332062-7be0-45b2-b02b-e4258bb756bd-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm\" (UID: \"91332062-7be0-45b2-b02b-e4258bb756bd\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:48:07.018888 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:07.018862    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czpjr\" (UniqueName: \"kubernetes.io/projected/91332062-7be0-45b2-b02b-e4258bb756bd-kube-api-access-czpjr\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm\" (UID: \"91332062-7be0-45b2-b02b-e4258bb756bd\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:48:07.098152 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:07.098119    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:48:07.235283 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:07.234750    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"]
Jul 28 17:48:07.237135 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:48:07.237106    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91332062_7be0_45b2_b02b_e4258bb756bd.slice/crio-0031f72179e49640c94cc18da5c8ada5583d6cd45e52101249fff576d8c619c1 WatchSource:0}: Error finding container 0031f72179e49640c94cc18da5c8ada5583d6cd45e52101249fff576d8c619c1: Status 404 returned error can't find the container with id 0031f72179e49640c94cc18da5c8ada5583d6cd45e52101249fff576d8c619c1
Jul 28 17:48:07.439106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:07.439001    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm" event={"ID":"91332062-7be0-45b2-b02b-e4258bb756bd","Type":"ContainerStarted","Data":"992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a"}
Jul 28 17:48:07.439106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:07.439052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm" event={"ID":"91332062-7be0-45b2-b02b-e4258bb756bd","Type":"ContainerStarted","Data":"0031f72179e49640c94cc18da5c8ada5583d6cd45e52101249fff576d8c619c1"}
Jul 28 17:48:07.439291 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:07.439113    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:48:07.458728 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:07.458676    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm" podStartSLOduration=1.458660603 podStartE2EDuration="1.458660603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:07.456936157 +0000 UTC m=+2275.686135266" watchObservedRunningTime="2026-07-28 17:48:07.458660603 +0000 UTC m=+2275.687859698"
Jul 28 17:48:38.445887 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:38.445854    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:48:48.914450 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:48.914417    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"]
Jul 28 17:48:58.446117 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:48:58.446047    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm" podUID="91332062-7be0-45b2-b02b-e4258bb756bd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:49:03.933988 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:03.933946    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm" podUID="91332062-7be0-45b2-b02b-e4258bb756bd" containerName="main" containerID="cri-o://992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a" gracePeriod=45
Jul 28 17:49:04.185312 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.185254    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:49:04.216628 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.216598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91332062-7be0-45b2-b02b-e4258bb756bd-tls-certs\") pod \"91332062-7be0-45b2-b02b-e4258bb756bd\" (UID: \"91332062-7be0-45b2-b02b-e4258bb756bd\") "
Jul 28 17:49:04.216820 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.216661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czpjr\" (UniqueName: \"kubernetes.io/projected/91332062-7be0-45b2-b02b-e4258bb756bd-kube-api-access-czpjr\") pod \"91332062-7be0-45b2-b02b-e4258bb756bd\" (UID: \"91332062-7be0-45b2-b02b-e4258bb756bd\") "
Jul 28 17:49:04.218804 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.218774    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91332062-7be0-45b2-b02b-e4258bb756bd-kube-api-access-czpjr" (OuterVolumeSpecName: "kube-api-access-czpjr") pod "91332062-7be0-45b2-b02b-e4258bb756bd" (UID: "91332062-7be0-45b2-b02b-e4258bb756bd"). InnerVolumeSpecName "kube-api-access-czpjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:49:04.218927 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.218783    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91332062-7be0-45b2-b02b-e4258bb756bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91332062-7be0-45b2-b02b-e4258bb756bd" (UID: "91332062-7be0-45b2-b02b-e4258bb756bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:49:04.317551 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.317522    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91332062-7be0-45b2-b02b-e4258bb756bd-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:49:04.317551 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.317548    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czpjr\" (UniqueName: \"kubernetes.io/projected/91332062-7be0-45b2-b02b-e4258bb756bd-kube-api-access-czpjr\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:49:04.729514 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.729476    2579 generic.go:356] "Generic (PLEG): container finished" podID="91332062-7be0-45b2-b02b-e4258bb756bd" containerID="992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a" exitCode=0
Jul 28 17:49:04.729691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.729519    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"
Jul 28 17:49:04.729691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.729559    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm" event={"ID":"91332062-7be0-45b2-b02b-e4258bb756bd","Type":"ContainerDied","Data":"992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a"}
Jul 28 17:49:04.729691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.729615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm" event={"ID":"91332062-7be0-45b2-b02b-e4258bb756bd","Type":"ContainerDied","Data":"0031f72179e49640c94cc18da5c8ada5583d6cd45e52101249fff576d8c619c1"}
Jul 28 17:49:04.729691 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.729637    2579 scope.go:117] "RemoveContainer" containerID="992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a"
Jul 28 17:49:04.738312 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.738294    2579 scope.go:117] "RemoveContainer" containerID="992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a"
Jul 28 17:49:04.738569 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:49:04.738550    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a\": container with ID starting with 992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a not found: ID does not exist" containerID="992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a"
Jul 28 17:49:04.738613 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.738582    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a"} err="failed to get container status \"992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a\": rpc error: code = NotFound desc = could not find container \"992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a\": container with ID starting with 992deccef88f967006d7156adafbc430dcb5026c423dec91f9688e869331863a not found: ID does not exist"
Jul 28 17:49:04.747535 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.747510    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"]
Jul 28 17:49:04.751583 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:04.751564    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schegsnjm"]
Jul 28 17:49:06.320094 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:06.320053    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91332062-7be0-45b2-b02b-e4258bb756bd" path="/var/lib/kubelet/pods/91332062-7be0-45b2-b02b-e4258bb756bd/volumes"
Jul 28 17:49:08.405375 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.405347    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"]
Jul 28 17:49:08.414342 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:49:08.414304    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 17:49:08.414342 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:49:08.414342 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:49:08.414342 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="ccc6dc47031f98f6018e703b745d3e053624df588b460dfc40900bb23768d80a" cmd=["/bin/sleep","15"]
Jul 28 17:49:08.414673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.414361    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c" message="" err=<
Jul 28 17:49:08.414673 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:49:08.414673 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:49:08.414673 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 17:49:08.414673 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:49:08.414378    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 17:49:08.414673 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 17:49:08.414673 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 17:49:08.414673 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c" podUID="b85a89a6-39ae-4176-aec9-c78c5334b391" containerName="main" containerID="cri-o://ccc6dc47031f98f6018e703b745d3e053624df588b460dfc40900bb23768d80a"
Jul 28 17:49:08.414673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.414484    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c" podUID="b85a89a6-39ae-4176-aec9-c78c5334b391" containerName="main" containerID="cri-o://ccc6dc47031f98f6018e703b745d3e053624df588b460dfc40900bb23768d80a" gracePeriod=60
Jul 28 17:49:08.417568 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.417544    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"]
Jul 28 17:49:08.749202 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.749174    2579 generic.go:356] "Generic (PLEG): container finished" podID="b85a89a6-39ae-4176-aec9-c78c5334b391" containerID="ccc6dc47031f98f6018e703b745d3e053624df588b460dfc40900bb23768d80a" exitCode=0
Jul 28 17:49:08.749353 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.749246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c" event={"ID":"b85a89a6-39ae-4176-aec9-c78c5334b391","Type":"ContainerDied","Data":"ccc6dc47031f98f6018e703b745d3e053624df588b460dfc40900bb23768d80a"}
Jul 28 17:49:08.770642 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.770621    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:49:08.854336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.854301    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-dshm\") pod \"b85a89a6-39ae-4176-aec9-c78c5334b391\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") "
Jul 28 17:49:08.854336 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.854341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85a89a6-39ae-4176-aec9-c78c5334b391-tls-certs\") pod \"b85a89a6-39ae-4176-aec9-c78c5334b391\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") "
Jul 28 17:49:08.854589 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.854389    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-model-cache\") pod \"b85a89a6-39ae-4176-aec9-c78c5334b391\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") "
Jul 28 17:49:08.854589 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.854415    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-home\") pod \"b85a89a6-39ae-4176-aec9-c78c5334b391\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") "
Jul 28 17:49:08.854589 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.854438    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v6cd\" (UniqueName: \"kubernetes.io/projected/b85a89a6-39ae-4176-aec9-c78c5334b391-kube-api-access-7v6cd\") pod \"b85a89a6-39ae-4176-aec9-c78c5334b391\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") "
Jul 28 17:49:08.854589 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.854482    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-tmp-dir\") pod \"b85a89a6-39ae-4176-aec9-c78c5334b391\" (UID: \"b85a89a6-39ae-4176-aec9-c78c5334b391\") "
Jul 28 17:49:08.854808 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.854624    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-model-cache" (OuterVolumeSpecName: "model-cache") pod "b85a89a6-39ae-4176-aec9-c78c5334b391" (UID: "b85a89a6-39ae-4176-aec9-c78c5334b391"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:49:08.854808 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.854739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-home" (OuterVolumeSpecName: "home") pod "b85a89a6-39ae-4176-aec9-c78c5334b391" (UID: "b85a89a6-39ae-4176-aec9-c78c5334b391"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:49:08.854903 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.854827    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b85a89a6-39ae-4176-aec9-c78c5334b391" (UID: "b85a89a6-39ae-4176-aec9-c78c5334b391"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:49:08.856482 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.856460    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-dshm" (OuterVolumeSpecName: "dshm") pod "b85a89a6-39ae-4176-aec9-c78c5334b391" (UID: "b85a89a6-39ae-4176-aec9-c78c5334b391"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:49:08.856612 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.856597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85a89a6-39ae-4176-aec9-c78c5334b391-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b85a89a6-39ae-4176-aec9-c78c5334b391" (UID: "b85a89a6-39ae-4176-aec9-c78c5334b391"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:49:08.856736 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.856718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85a89a6-39ae-4176-aec9-c78c5334b391-kube-api-access-7v6cd" (OuterVolumeSpecName: "kube-api-access-7v6cd") pod "b85a89a6-39ae-4176-aec9-c78c5334b391" (UID: "b85a89a6-39ae-4176-aec9-c78c5334b391"). InnerVolumeSpecName "kube-api-access-7v6cd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:49:08.955564 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.955486    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:49:08.955564 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.955514    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:49:08.955564 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.955524    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v6cd\" (UniqueName: \"kubernetes.io/projected/b85a89a6-39ae-4176-aec9-c78c5334b391-kube-api-access-7v6cd\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:49:08.955564 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.955535    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:49:08.955564 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.955545    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b85a89a6-39ae-4176-aec9-c78c5334b391-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:49:08.955564 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:08.955553    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b85a89a6-39ae-4176-aec9-c78c5334b391-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:49:09.754973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:09.754937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c" event={"ID":"b85a89a6-39ae-4176-aec9-c78c5334b391","Type":"ContainerDied","Data":"957fce5a0b9afe5100395cf4a10181e62d458f6b7576caa7416702c6839b666f"}
Jul 28 17:49:09.754973 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:09.754958    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"
Jul 28 17:49:09.755551 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:09.755006    2579 scope.go:117] "RemoveContainer" containerID="ccc6dc47031f98f6018e703b745d3e053624df588b460dfc40900bb23768d80a"
Jul 28 17:49:09.777193 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:09.777165    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"]
Jul 28 17:49:09.781458 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:09.781432    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969rdq6c"]
Jul 28 17:49:10.320034 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:10.319996    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b85a89a6-39ae-4176-aec9-c78c5334b391" path="/var/lib/kubelet/pods/b85a89a6-39ae-4176-aec9-c78c5334b391/volumes"
Jul 28 17:49:12.218929 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:12.218889    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h" podUID="d34cc5f4-a485-4153-bb14-b4404f9eb92b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:49:22.219327 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:22.219281    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h" podUID="d34cc5f4-a485-4153-bb14-b4404f9eb92b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:49:23.437897 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.437858    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h" podUID="d34cc5f4-a485-4153-bb14-b4404f9eb92b" containerName="main" containerID="cri-o://82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050" gracePeriod=45
Jul 28 17:49:23.690449 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.690394    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:49:23.787791 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.787744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmbqj\" (UniqueName: \"kubernetes.io/projected/d34cc5f4-a485-4153-bb14-b4404f9eb92b-kube-api-access-jmbqj\") pod \"d34cc5f4-a485-4153-bb14-b4404f9eb92b\" (UID: \"d34cc5f4-a485-4153-bb14-b4404f9eb92b\") "
Jul 28 17:49:23.787992 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.787813    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34cc5f4-a485-4153-bb14-b4404f9eb92b-tls-certs\") pod \"d34cc5f4-a485-4153-bb14-b4404f9eb92b\" (UID: \"d34cc5f4-a485-4153-bb14-b4404f9eb92b\") "
Jul 28 17:49:23.789854 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.789828    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d34cc5f4-a485-4153-bb14-b4404f9eb92b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d34cc5f4-a485-4153-bb14-b4404f9eb92b" (UID: "d34cc5f4-a485-4153-bb14-b4404f9eb92b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:49:23.789940 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.789866    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34cc5f4-a485-4153-bb14-b4404f9eb92b-kube-api-access-jmbqj" (OuterVolumeSpecName: "kube-api-access-jmbqj") pod "d34cc5f4-a485-4153-bb14-b4404f9eb92b" (UID: "d34cc5f4-a485-4153-bb14-b4404f9eb92b"). InnerVolumeSpecName "kube-api-access-jmbqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:49:23.833234 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.833200    2579 generic.go:356] "Generic (PLEG): container finished" podID="d34cc5f4-a485-4153-bb14-b4404f9eb92b" containerID="82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050" exitCode=0
Jul 28 17:49:23.833398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.833242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h" event={"ID":"d34cc5f4-a485-4153-bb14-b4404f9eb92b","Type":"ContainerDied","Data":"82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050"}
Jul 28 17:49:23.833398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.833255    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"
Jul 28 17:49:23.833398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.833320    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h" event={"ID":"d34cc5f4-a485-4153-bb14-b4404f9eb92b","Type":"ContainerDied","Data":"1f418c9c0b74827710a4ec89e1e58303cc115881743dd053c222f929755f0fa6"}
Jul 28 17:49:23.833398 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.833349    2579 scope.go:117] "RemoveContainer" containerID="82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050"
Jul 28 17:49:23.842159 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.842142    2579 scope.go:117] "RemoveContainer" containerID="82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050"
Jul 28 17:49:23.842423 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:49:23.842405    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050\": container with ID starting with 82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050 not found: ID does not exist" containerID="82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050"
Jul 28 17:49:23.842473 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.842432    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050"} err="failed to get container status \"82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050\": rpc error: code = NotFound desc = could not find container \"82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050\": container with ID starting with 82099885b8351213e70a24133766bce505e371303783f15019e7981742c2e050 not found: ID does not exist"
Jul 28 17:49:23.856165 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.856134    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"]
Jul 28 17:49:23.862661 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.862630    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schebtf7h"]
Jul 28 17:49:23.888976 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.888948    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmbqj\" (UniqueName: \"kubernetes.io/projected/d34cc5f4-a485-4153-bb14-b4404f9eb92b-kube-api-access-jmbqj\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:49:23.889064 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:23.888977    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d34cc5f4-a485-4153-bb14-b4404f9eb92b-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:49:24.320358 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:24.320326    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d34cc5f4-a485-4153-bb14-b4404f9eb92b" path="/var/lib/kubelet/pods/d34cc5f4-a485-4153-bb14-b4404f9eb92b/volumes"
Jul 28 17:49:33.152017 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.151987    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"]
Jul 28 17:49:33.152425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.152370    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85a89a6-39ae-4176-aec9-c78c5334b391" containerName="main"
Jul 28 17:49:33.152425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.152382    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85a89a6-39ae-4176-aec9-c78c5334b391" containerName="main"
Jul 28 17:49:33.152425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.152393    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34cc5f4-a485-4153-bb14-b4404f9eb92b" containerName="main"
Jul 28 17:49:33.152425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.152398    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34cc5f4-a485-4153-bb14-b4404f9eb92b" containerName="main"
Jul 28 17:49:33.152425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.152414    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91332062-7be0-45b2-b02b-e4258bb756bd" containerName="main"
Jul 28 17:49:33.152425 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.152420    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="91332062-7be0-45b2-b02b-e4258bb756bd" containerName="main"
Jul 28 17:49:33.152641 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.152485    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="91332062-7be0-45b2-b02b-e4258bb756bd" containerName="main"
Jul 28 17:49:33.152641 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.152493    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85a89a6-39ae-4176-aec9-c78c5334b391" containerName="main"
Jul 28 17:49:33.152641 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.152501    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d34cc5f4-a485-4153-bb14-b4404f9eb92b" containerName="main"
Jul 28 17:49:33.156947 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.156931    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.160182 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.160158    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 28 17:49:33.160292 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.160281    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 28 17:49:33.161229 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.161211    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 28 17:49:33.161464 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.161305    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:49:33.161464 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.161334    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-dm4n7\""
Jul 28 17:49:33.161464 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.161394    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"seaweedfs-s3-creds\""
Jul 28 17:49:33.170219 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.170194    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"]
Jul 28 17:49:33.271228 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.271199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7c1945b-7013-4e44-94b2-8219641009a9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.271408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.271233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkmrl\" (UniqueName: \"kubernetes.io/projected/f7c1945b-7013-4e44-94b2-8219641009a9-kube-api-access-qkmrl\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.271408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.271274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c1945b-7013-4e44-94b2-8219641009a9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.271408 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.271326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.271521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.271424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.271521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.271471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.271521 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.271501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.271620 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.271526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.365406 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.365374    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"]
Jul 28 17:49:33.368731 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.368712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:49:33.371711 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.371688    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-jkm62\""
Jul 28 17:49:33.372025 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.372004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.372143 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.372040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.372143 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.372115    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.372239 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.372209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7c1945b-7013-4e44-94b2-8219641009a9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.372285 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.372252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkmrl\" (UniqueName: \"kubernetes.io/projected/f7c1945b-7013-4e44-94b2-8219641009a9-kube-api-access-qkmrl\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.372338 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.372304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c1945b-7013-4e44-94b2-8219641009a9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.372409 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.372386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.372466 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.372455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.372563 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.372545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.372835 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.372814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.373019 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.373001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.373115 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.373007    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7c1945b-7013-4e44-94b2-8219641009a9-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.373186 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.373129    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.374549 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.374528    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.374872 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.374854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c1945b-7013-4e44-94b2-8219641009a9-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.382642 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.382616    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"]
Jul 28 17:49:33.385335 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.385309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkmrl\" (UniqueName: \"kubernetes.io/projected/f7c1945b-7013-4e44-94b2-8219641009a9-kube-api-access-qkmrl\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.467332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.467248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:33.473674 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.473635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5r9m\" (UniqueName: \"kubernetes.io/projected/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-kube-api-access-l5r9m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd\" (UID: \"3ef77a4a-3fa8-467d-bbcc-702bda8e5129\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:49:33.473787 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.473719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd\" (UID: \"3ef77a4a-3fa8-467d-bbcc-702bda8e5129\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:49:33.574676 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.574640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5r9m\" (UniqueName: \"kubernetes.io/projected/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-kube-api-access-l5r9m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd\" (UID: \"3ef77a4a-3fa8-467d-bbcc-702bda8e5129\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:49:33.574853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.574710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd\" (UID: \"3ef77a4a-3fa8-467d-bbcc-702bda8e5129\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:49:33.577255 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.577229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd\" (UID: \"3ef77a4a-3fa8-467d-bbcc-702bda8e5129\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:49:33.583672 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.583650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5r9m\" (UniqueName: \"kubernetes.io/projected/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-kube-api-access-l5r9m\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd\" (UID: \"3ef77a4a-3fa8-467d-bbcc-702bda8e5129\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:49:33.606100 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.605956    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"]
Jul 28 17:49:33.608336 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:49:33.608314    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7c1945b_7013_4e44_94b2_8219641009a9.slice/crio-d9c9164278823e65912f25d089666038e99ec73db759a239c48128f013b20ae8 WatchSource:0}: Error finding container d9c9164278823e65912f25d089666038e99ec73db759a239c48128f013b20ae8: Status 404 returned error can't find the container with id d9c9164278823e65912f25d089666038e99ec73db759a239c48128f013b20ae8
Jul 28 17:49:33.695458 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.695429    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:49:33.835708 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.835680    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"]
Jul 28 17:49:33.837089 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:49:33.837036    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef77a4a_3fa8_467d_bbcc_702bda8e5129.slice/crio-3c2fc6d3fab9d3e8e3091fb26f2950f397e2dca331c6fbfd5caa1eed2bdb4510 WatchSource:0}: Error finding container 3c2fc6d3fab9d3e8e3091fb26f2950f397e2dca331c6fbfd5caa1eed2bdb4510: Status 404 returned error can't find the container with id 3c2fc6d3fab9d3e8e3091fb26f2950f397e2dca331c6fbfd5caa1eed2bdb4510
Jul 28 17:49:33.895155 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.895117    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd" event={"ID":"3ef77a4a-3fa8-467d-bbcc-702bda8e5129","Type":"ContainerStarted","Data":"3c2fc6d3fab9d3e8e3091fb26f2950f397e2dca331c6fbfd5caa1eed2bdb4510"}
Jul 28 17:49:33.896825 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.896801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" event={"ID":"f7c1945b-7013-4e44-94b2-8219641009a9","Type":"ContainerStarted","Data":"6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37"}
Jul 28 17:49:33.896941 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:33.896842    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" event={"ID":"f7c1945b-7013-4e44-94b2-8219641009a9","Type":"ContainerStarted","Data":"d9c9164278823e65912f25d089666038e99ec73db759a239c48128f013b20ae8"}
Jul 28 17:49:34.904735 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:34.904672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd" event={"ID":"3ef77a4a-3fa8-467d-bbcc-702bda8e5129","Type":"ContainerStarted","Data":"fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca"}
Jul 28 17:49:34.905332 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:34.904779    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:49:34.924578 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:34.924476    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd" podStartSLOduration=1.924460751 podStartE2EDuration="1.924460751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:49:34.922933881 +0000 UTC m=+2363.152133013" watchObservedRunningTime="2026-07-28 17:49:34.924460751 +0000 UTC m=+2363.153659846"
Jul 28 17:49:38.926746 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:38.926714    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7c1945b-7013-4e44-94b2-8219641009a9" containerID="6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37" exitCode=0
Jul 28 17:49:38.927201 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:38.926792    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" event={"ID":"f7c1945b-7013-4e44-94b2-8219641009a9","Type":"ContainerDied","Data":"6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37"}
Jul 28 17:49:39.934925 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:39.934891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" event={"ID":"f7c1945b-7013-4e44-94b2-8219641009a9","Type":"ContainerStarted","Data":"ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637"}
Jul 28 17:49:39.960622 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:39.960550    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" podStartSLOduration=2.255849044 podStartE2EDuration="6.96053257s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.704683526s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:49:39.958633377 +0000 UTC m=+2368.187832473" watchObservedRunningTime="2026-07-28 17:49:39.96053257 +0000 UTC m=+2368.189731665"
Jul 28 17:49:40.467659 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:40.467616    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:43.467994 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:43.467951    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:49:43.469717 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:43.469687    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 28 17:49:53.468349 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:49:53.468309    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 28 17:50:03.468028 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:50:03.467979    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 28 17:50:05.913641 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:50:05.913609    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:50:12.394057 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:50:12.394028    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:50:12.399399 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:50:12.399376    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:50:13.468043 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:50:13.468001    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 28 17:50:23.468430 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:50:23.468384    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 28 17:50:33.467955 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:50:33.467909    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 28 17:50:43.467912 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:50:43.467869    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/health\": dial tcp 10.132.0.62:8000: connect: connection refused"
Jul 28 17:50:53.477555 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:50:53.477520    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:50:53.485621 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:50:53.485602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:51:17.658942 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:17.658906    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"]
Jul 28 17:51:17.663613 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:17.663585    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"]
Jul 28 17:51:25.913108 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:25.913046    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd" podUID="3ef77a4a-3fa8-467d-bbcc-702bda8e5129" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:51:32.678769 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:32.678705    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd" podUID="3ef77a4a-3fa8-467d-bbcc-702bda8e5129" containerName="main" containerID="cri-o://fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca" gracePeriod=45
Jul 28 17:51:32.685884 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:32.685852    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main" containerID="cri-o://ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637" gracePeriod=45
Jul 28 17:51:32.932352 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:32.932275    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:51:33.115388 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.115352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5r9m\" (UniqueName: \"kubernetes.io/projected/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-kube-api-access-l5r9m\") pod \"3ef77a4a-3fa8-467d-bbcc-702bda8e5129\" (UID: \"3ef77a4a-3fa8-467d-bbcc-702bda8e5129\") "
Jul 28 17:51:33.115388 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.115394    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-tls-certs\") pod \"3ef77a4a-3fa8-467d-bbcc-702bda8e5129\" (UID: \"3ef77a4a-3fa8-467d-bbcc-702bda8e5129\") "
Jul 28 17:51:33.117628 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.117595    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ef77a4a-3fa8-467d-bbcc-702bda8e5129" (UID: "3ef77a4a-3fa8-467d-bbcc-702bda8e5129"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:51:33.117751 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.117643    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-kube-api-access-l5r9m" (OuterVolumeSpecName: "kube-api-access-l5r9m") pod "3ef77a4a-3fa8-467d-bbcc-702bda8e5129" (UID: "3ef77a4a-3fa8-467d-bbcc-702bda8e5129"). InnerVolumeSpecName "kube-api-access-l5r9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:51:33.216826 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.216798    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:51:33.216826 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.216827    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5r9m\" (UniqueName: \"kubernetes.io/projected/3ef77a4a-3fa8-467d-bbcc-702bda8e5129-kube-api-access-l5r9m\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:51:33.329209 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.329182    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:51:33.518445 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c1945b-7013-4e44-94b2-8219641009a9-tls-certs\") pod \"f7c1945b-7013-4e44-94b2-8219641009a9\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") "
Jul 28 17:51:33.518445 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518411    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-kserve-provision-location\") pod \"f7c1945b-7013-4e44-94b2-8219641009a9\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") "
Jul 28 17:51:33.518673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518457    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-home\") pod \"f7c1945b-7013-4e44-94b2-8219641009a9\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") "
Jul 28 17:51:33.518673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkmrl\" (UniqueName: \"kubernetes.io/projected/f7c1945b-7013-4e44-94b2-8219641009a9-kube-api-access-qkmrl\") pod \"f7c1945b-7013-4e44-94b2-8219641009a9\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") "
Jul 28 17:51:33.518673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518515    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7c1945b-7013-4e44-94b2-8219641009a9-cabundle-cert\") pod \"f7c1945b-7013-4e44-94b2-8219641009a9\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") "
Jul 28 17:51:33.518673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518555    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-model-cache\") pod \"f7c1945b-7013-4e44-94b2-8219641009a9\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") "
Jul 28 17:51:33.518673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518582    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-tmp-dir\") pod \"f7c1945b-7013-4e44-94b2-8219641009a9\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") "
Jul 28 17:51:33.518673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518613    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-dshm\") pod \"f7c1945b-7013-4e44-94b2-8219641009a9\" (UID: \"f7c1945b-7013-4e44-94b2-8219641009a9\") "
Jul 28 17:51:33.518986 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518910    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7c1945b-7013-4e44-94b2-8219641009a9" (UID: "f7c1945b-7013-4e44-94b2-8219641009a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:33.518986 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-home" (OuterVolumeSpecName: "home") pod "f7c1945b-7013-4e44-94b2-8219641009a9" (UID: "f7c1945b-7013-4e44-94b2-8219641009a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:33.519122 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.518977    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7c1945b-7013-4e44-94b2-8219641009a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f7c1945b-7013-4e44-94b2-8219641009a9" (UID: "f7c1945b-7013-4e44-94b2-8219641009a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:51:33.520936 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.520904    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7c1945b-7013-4e44-94b2-8219641009a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7c1945b-7013-4e44-94b2-8219641009a9" (UID: "f7c1945b-7013-4e44-94b2-8219641009a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:51:33.521115 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.521095    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-dshm" (OuterVolumeSpecName: "dshm") pod "f7c1945b-7013-4e44-94b2-8219641009a9" (UID: "f7c1945b-7013-4e44-94b2-8219641009a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:33.521203 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.521121    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7c1945b-7013-4e44-94b2-8219641009a9-kube-api-access-qkmrl" (OuterVolumeSpecName: "kube-api-access-qkmrl") pod "f7c1945b-7013-4e44-94b2-8219641009a9" (UID: "f7c1945b-7013-4e44-94b2-8219641009a9"). InnerVolumeSpecName "kube-api-access-qkmrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:51:33.525982 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.525957    2579 generic.go:356] "Generic (PLEG): container finished" podID="3ef77a4a-3fa8-467d-bbcc-702bda8e5129" containerID="fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca" exitCode=0
Jul 28 17:51:33.526072 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.525998    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"
Jul 28 17:51:33.526072 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.526036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd" event={"ID":"3ef77a4a-3fa8-467d-bbcc-702bda8e5129","Type":"ContainerDied","Data":"fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca"}
Jul 28 17:51:33.526235 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.526216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd" event={"ID":"3ef77a4a-3fa8-467d-bbcc-702bda8e5129","Type":"ContainerDied","Data":"3c2fc6d3fab9d3e8e3091fb26f2950f397e2dca331c6fbfd5caa1eed2bdb4510"}
Jul 28 17:51:33.526298 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.526251    2579 scope.go:117] "RemoveContainer" containerID="fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca"
Jul 28 17:51:33.528348 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.528324    2579 generic.go:356] "Generic (PLEG): container finished" podID="f7c1945b-7013-4e44-94b2-8219641009a9" containerID="ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637" exitCode=0
Jul 28 17:51:33.528451 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.528395    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" event={"ID":"f7c1945b-7013-4e44-94b2-8219641009a9","Type":"ContainerDied","Data":"ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637"}
Jul 28 17:51:33.528451 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.528433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4" event={"ID":"f7c1945b-7013-4e44-94b2-8219641009a9","Type":"ContainerDied","Data":"d9c9164278823e65912f25d089666038e99ec73db759a239c48128f013b20ae8"}
Jul 28 17:51:33.528570 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.528501    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"
Jul 28 17:51:33.531185 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.531144    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7c1945b-7013-4e44-94b2-8219641009a9" (UID: "f7c1945b-7013-4e44-94b2-8219641009a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:33.536224 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.536201    2579 scope.go:117] "RemoveContainer" containerID="fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca"
Jul 28 17:51:33.536528 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:51:33.536507    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca\": container with ID starting with fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca not found: ID does not exist" containerID="fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca"
Jul 28 17:51:33.536591 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.536540    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca"} err="failed to get container status \"fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca\": rpc error: code = NotFound desc = could not find container \"fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca\": container with ID starting with fd9f142775c1d70b93ffb393e9b4326b2d85c0130fd05adaa2914df47de321ca not found: ID does not exist"
Jul 28 17:51:33.536591 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.536563    2579 scope.go:117] "RemoveContainer" containerID="ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637"
Jul 28 17:51:33.545925 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.545906    2579 scope.go:117] "RemoveContainer" containerID="6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37"
Jul 28 17:51:33.550679 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.550657    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"]
Jul 28 17:51:33.553180 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.553161    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schelbrtd"]
Jul 28 17:51:33.555455 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.555439    2579 scope.go:117] "RemoveContainer" containerID="ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637"
Jul 28 17:51:33.555695 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:51:33.555670    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637\": container with ID starting with ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637 not found: ID does not exist" containerID="ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637"
Jul 28 17:51:33.555755 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.555706    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637"} err="failed to get container status \"ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637\": rpc error: code = NotFound desc = could not find container \"ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637\": container with ID starting with ea02d7218dd9c800c84b1912eac5b6f77c2f776fbf4cd57ace79ae7d7119c637 not found: ID does not exist"
Jul 28 17:51:33.555755 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.555729    2579 scope.go:117] "RemoveContainer" containerID="6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37"
Jul 28 17:51:33.556000 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:51:33.555982    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37\": container with ID starting with 6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37 not found: ID does not exist" containerID="6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37"
Jul 28 17:51:33.556044 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.556008    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37"} err="failed to get container status \"6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37\": rpc error: code = NotFound desc = could not find container \"6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37\": container with ID starting with 6f5343f3e07c859ac62fab8e9e3626a6c12cb5271a189de7af7e1038ba4f8a37 not found: ID does not exist"
Jul 28 17:51:33.585992 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.585968    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f7c1945b-7013-4e44-94b2-8219641009a9" (UID: "f7c1945b-7013-4e44-94b2-8219641009a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:33.619772 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.619751    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:51:33.619772 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.619771    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:51:33.619902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.619783    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:51:33.619902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.619790    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7c1945b-7013-4e44-94b2-8219641009a9-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:51:33.619902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.619798    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:51:33.619902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.619806    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7c1945b-7013-4e44-94b2-8219641009a9-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:51:33.619902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.619815    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkmrl\" (UniqueName: \"kubernetes.io/projected/f7c1945b-7013-4e44-94b2-8219641009a9-kube-api-access-qkmrl\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:51:33.619902 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.619823    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f7c1945b-7013-4e44-94b2-8219641009a9-cabundle-cert\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:51:33.854110 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.854062    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"]
Jul 28 17:51:33.862341 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:33.862316    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-646b985775-qczd4"]
Jul 28 17:51:34.320769 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:34.320731    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ef77a4a-3fa8-467d-bbcc-702bda8e5129" path="/var/lib/kubelet/pods/3ef77a4a-3fa8-467d-bbcc-702bda8e5129/volumes"
Jul 28 17:51:34.321271 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:34.321251    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" path="/var/lib/kubelet/pods/f7c1945b-7013-4e44-94b2-8219641009a9/volumes"
Jul 28 17:51:55.901140 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.901103    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"]
Jul 28 17:51:55.901660 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.901474    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ef77a4a-3fa8-467d-bbcc-702bda8e5129" containerName="main"
Jul 28 17:51:55.901660 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.901486    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ef77a4a-3fa8-467d-bbcc-702bda8e5129" containerName="main"
Jul 28 17:51:55.901660 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.901510    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="storage-initializer"
Jul 28 17:51:55.901660 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.901516    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="storage-initializer"
Jul 28 17:51:55.901660 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.901525    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main"
Jul 28 17:51:55.901660 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.901531    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main"
Jul 28 17:51:55.901660 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.901582    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ef77a4a-3fa8-467d-bbcc-702bda8e5129" containerName="main"
Jul 28 17:51:55.901660 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.901592    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7c1945b-7013-4e44-94b2-8219641009a9" containerName="main"
Jul 28 17:51:55.903942 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.903924    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:55.907144 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.907116    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-s4p5k\""
Jul 28 17:51:55.907286 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.907167    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"seaweedfs-s3-creds\""
Jul 28 17:51:55.908032 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.908011    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 28 17:51:55.908032 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.908033    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:51:55.908267 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.908018    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 28 17:51:55.909106 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.909072    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 28 17:51:55.928301 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:55.928275    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"]
Jul 28 17:51:56.002588 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.002553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29cb07da-1c23-428b-bfdc-f5558df04093-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.002755 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.002598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.002755 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.002619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.002755 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.002719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.002755 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.002752    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29cb07da-1c23-428b-bfdc-f5558df04093-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.002900 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.002781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.002900 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.002814    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzx8s\" (UniqueName: \"kubernetes.io/projected/29cb07da-1c23-428b-bfdc-f5558df04093-kube-api-access-hzx8s\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.002900 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.002838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104293 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29cb07da-1c23-428b-bfdc-f5558df04093-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104328    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104388    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104664 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29cb07da-1c23-428b-bfdc-f5558df04093-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104664 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104664 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzx8s\" (UniqueName: \"kubernetes.io/projected/29cb07da-1c23-428b-bfdc-f5558df04093-kube-api-access-hzx8s\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104851 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104916 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104899    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.104998 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.104965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.105184 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.105163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.105311 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.105292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29cb07da-1c23-428b-bfdc-f5558df04093-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.106697 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.106680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.106793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.106777    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29cb07da-1c23-428b-bfdc-f5558df04093-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.117564 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.117541    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzx8s\" (UniqueName: \"kubernetes.io/projected/29cb07da-1c23-428b-bfdc-f5558df04093-kube-api-access-hzx8s\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.216994 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.216915    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:51:56.278509 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.278480    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"]
Jul 28 17:51:56.281690 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.281665    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:51:56.285413 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.285379    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-bqq5s\""
Jul 28 17:51:56.304437 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.304410    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"]
Jul 28 17:51:56.382360 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.380844    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"]
Jul 28 17:51:56.384936 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:51:56.384906    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cb07da_1c23_428b_bfdc_f5558df04093.slice/crio-d684fba86f052444b6a5c809f243c0745d1f22cb6b1d27703e1be1010051f06c WatchSource:0}: Error finding container d684fba86f052444b6a5c809f243c0745d1f22cb6b1d27703e1be1010051f06c: Status 404 returned error can't find the container with id d684fba86f052444b6a5c809f243c0745d1f22cb6b1d27703e1be1010051f06c
Jul 28 17:51:56.408547 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.408516    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348018a9-0918-47d5-b0f6-8d512138692c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd\" (UID: \"348018a9-0918-47d5-b0f6-8d512138692c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:51:56.408841 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.408813    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2kl\" (UniqueName: \"kubernetes.io/projected/348018a9-0918-47d5-b0f6-8d512138692c-kube-api-access-pm2kl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd\" (UID: \"348018a9-0918-47d5-b0f6-8d512138692c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:51:56.510188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.510110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348018a9-0918-47d5-b0f6-8d512138692c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd\" (UID: \"348018a9-0918-47d5-b0f6-8d512138692c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:51:56.510188 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.510168    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2kl\" (UniqueName: \"kubernetes.io/projected/348018a9-0918-47d5-b0f6-8d512138692c-kube-api-access-pm2kl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd\" (UID: \"348018a9-0918-47d5-b0f6-8d512138692c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:51:56.512572 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.512544    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348018a9-0918-47d5-b0f6-8d512138692c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd\" (UID: \"348018a9-0918-47d5-b0f6-8d512138692c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:51:56.520071 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.520044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2kl\" (UniqueName: \"kubernetes.io/projected/348018a9-0918-47d5-b0f6-8d512138692c-kube-api-access-pm2kl\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd\" (UID: \"348018a9-0918-47d5-b0f6-8d512138692c\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:51:56.594557 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.594524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:51:56.642560 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.642403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" event={"ID":"29cb07da-1c23-428b-bfdc-f5558df04093","Type":"ContainerStarted","Data":"6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431"}
Jul 28 17:51:56.642560 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.642469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" event={"ID":"29cb07da-1c23-428b-bfdc-f5558df04093","Type":"ContainerStarted","Data":"d684fba86f052444b6a5c809f243c0745d1f22cb6b1d27703e1be1010051f06c"}
Jul 28 17:51:56.729011 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:56.728985    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"]
Jul 28 17:51:56.731115 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:51:56.731087    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348018a9_0918_47d5_b0f6_8d512138692c.slice/crio-8287a6f07fa446aba927198f9cb7cee62d20ac57a73c61fd0a5cab96c58e8cb2 WatchSource:0}: Error finding container 8287a6f07fa446aba927198f9cb7cee62d20ac57a73c61fd0a5cab96c58e8cb2: Status 404 returned error can't find the container with id 8287a6f07fa446aba927198f9cb7cee62d20ac57a73c61fd0a5cab96c58e8cb2
Jul 28 17:51:57.650098 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:57.650044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd" event={"ID":"348018a9-0918-47d5-b0f6-8d512138692c","Type":"ContainerStarted","Data":"363554868bfcc4c4a54849514976528b0869fdf9b98f0e32c0fa913551fc5c69"}
Jul 28 17:51:57.650648 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:57.650134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd" event={"ID":"348018a9-0918-47d5-b0f6-8d512138692c","Type":"ContainerStarted","Data":"8287a6f07fa446aba927198f9cb7cee62d20ac57a73c61fd0a5cab96c58e8cb2"}
Jul 28 17:51:57.650648 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:57.650340    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:51:57.689811 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:51:57.689753    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd" podStartSLOduration=1.68973191 podStartE2EDuration="1.68973191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:51:57.686240779 +0000 UTC m=+2505.915439875" watchObservedRunningTime="2026-07-28 17:51:57.68973191 +0000 UTC m=+2505.918931007"
Jul 28 17:52:02.676630 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:02.676549    2579 generic.go:356] "Generic (PLEG): container finished" podID="29cb07da-1c23-428b-bfdc-f5558df04093" containerID="6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431" exitCode=0
Jul 28 17:52:02.676984 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:02.676623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" event={"ID":"29cb07da-1c23-428b-bfdc-f5558df04093","Type":"ContainerDied","Data":"6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431"}
Jul 28 17:52:03.684187 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:03.684150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" event={"ID":"29cb07da-1c23-428b-bfdc-f5558df04093","Type":"ContainerStarted","Data":"e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85"}
Jul 28 17:52:03.709180 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:03.709119    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" podStartSLOduration=2.738378104 podStartE2EDuration="8.709103575s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.970725471s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:51:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:03.708284107 +0000 UTC m=+2511.937483205" watchObservedRunningTime="2026-07-28 17:52:03.709103575 +0000 UTC m=+2511.938302668"
Jul 28 17:52:04.217218 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:04.217177    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:52:06.217545 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:06.217508    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:52:06.219013 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:06.218968    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 28 17:52:16.217689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:16.217644    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 28 17:52:26.217388 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:26.217342    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 28 17:52:28.657874 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:28.657842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:52:36.218296 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:36.218251    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 28 17:52:46.218029 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:46.217985    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 28 17:52:56.218116 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:52:56.218051    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 28 17:53:06.217527 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:06.217488    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 28 17:53:16.227701 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:16.227666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:53:16.235444 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:16.235424    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:53:17.891322 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:17.891278    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"]
Jul 28 17:53:17.896162 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:17.896133    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"]
Jul 28 17:53:18.657270 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:18.657234    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd" podUID="348018a9-0918-47d5-b0f6-8d512138692c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:53:28.657621 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:28.657575    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd" podUID="348018a9-0918-47d5-b0f6-8d512138692c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:53:32.910919 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:32.910857    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd" podUID="348018a9-0918-47d5-b0f6-8d512138692c" containerName="main" containerID="cri-o://363554868bfcc4c4a54849514976528b0869fdf9b98f0e32c0fa913551fc5c69" gracePeriod=45
Jul 28 17:53:33.076831 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.076784    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main" containerID="cri-o://e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85" gracePeriod=45
Jul 28 17:53:33.132523 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.132491    2579 generic.go:356] "Generic (PLEG): container finished" podID="348018a9-0918-47d5-b0f6-8d512138692c" containerID="363554868bfcc4c4a54849514976528b0869fdf9b98f0e32c0fa913551fc5c69" exitCode=0
Jul 28 17:53:33.132687 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.132558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd" event={"ID":"348018a9-0918-47d5-b0f6-8d512138692c","Type":"ContainerDied","Data":"363554868bfcc4c4a54849514976528b0869fdf9b98f0e32c0fa913551fc5c69"}
Jul 28 17:53:33.166059 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.166002    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:53:33.229294 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.229246    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": dial tcp 10.132.0.64:8000: connect: connection refused"
Jul 28 17:53:33.291097 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.291045    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348018a9-0918-47d5-b0f6-8d512138692c-tls-certs\") pod \"348018a9-0918-47d5-b0f6-8d512138692c\" (UID: \"348018a9-0918-47d5-b0f6-8d512138692c\") "
Jul 28 17:53:33.291254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.291107    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm2kl\" (UniqueName: \"kubernetes.io/projected/348018a9-0918-47d5-b0f6-8d512138692c-kube-api-access-pm2kl\") pod \"348018a9-0918-47d5-b0f6-8d512138692c\" (UID: \"348018a9-0918-47d5-b0f6-8d512138692c\") "
Jul 28 17:53:33.293254 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.293223    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348018a9-0918-47d5-b0f6-8d512138692c-kube-api-access-pm2kl" (OuterVolumeSpecName: "kube-api-access-pm2kl") pod "348018a9-0918-47d5-b0f6-8d512138692c" (UID: "348018a9-0918-47d5-b0f6-8d512138692c"). InnerVolumeSpecName "kube-api-access-pm2kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:33.293388 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.293324    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348018a9-0918-47d5-b0f6-8d512138692c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "348018a9-0918-47d5-b0f6-8d512138692c" (UID: "348018a9-0918-47d5-b0f6-8d512138692c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:33.392587 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.392551    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm2kl\" (UniqueName: \"kubernetes.io/projected/348018a9-0918-47d5-b0f6-8d512138692c-kube-api-access-pm2kl\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.392587 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.392579    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/348018a9-0918-47d5-b0f6-8d512138692c-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.612717 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.612694    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:53:33.694673 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.694642    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzx8s\" (UniqueName: \"kubernetes.io/projected/29cb07da-1c23-428b-bfdc-f5558df04093-kube-api-access-hzx8s\") pod \"29cb07da-1c23-428b-bfdc-f5558df04093\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") "
Jul 28 17:53:33.694853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.694690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-model-cache\") pod \"29cb07da-1c23-428b-bfdc-f5558df04093\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") "
Jul 28 17:53:33.694853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.694714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-home\") pod \"29cb07da-1c23-428b-bfdc-f5558df04093\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") "
Jul 28 17:53:33.694853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.694735    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29cb07da-1c23-428b-bfdc-f5558df04093-cabundle-cert\") pod \"29cb07da-1c23-428b-bfdc-f5558df04093\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") "
Jul 28 17:53:33.694853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.694751    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-tmp-dir\") pod \"29cb07da-1c23-428b-bfdc-f5558df04093\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") "
Jul 28 17:53:33.694853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.694770    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29cb07da-1c23-428b-bfdc-f5558df04093-tls-certs\") pod \"29cb07da-1c23-428b-bfdc-f5558df04093\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") "
Jul 28 17:53:33.694853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.694805    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-dshm\") pod \"29cb07da-1c23-428b-bfdc-f5558df04093\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") "
Jul 28 17:53:33.694853 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.694838    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-kserve-provision-location\") pod \"29cb07da-1c23-428b-bfdc-f5558df04093\" (UID: \"29cb07da-1c23-428b-bfdc-f5558df04093\") "
Jul 28 17:53:33.695252 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.695016    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-model-cache" (OuterVolumeSpecName: "model-cache") pod "29cb07da-1c23-428b-bfdc-f5558df04093" (UID: "29cb07da-1c23-428b-bfdc-f5558df04093"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:33.695252 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.695175    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-home" (OuterVolumeSpecName: "home") pod "29cb07da-1c23-428b-bfdc-f5558df04093" (UID: "29cb07da-1c23-428b-bfdc-f5558df04093"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:33.695355 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.695187    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29cb07da-1c23-428b-bfdc-f5558df04093-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "29cb07da-1c23-428b-bfdc-f5558df04093" (UID: "29cb07da-1c23-428b-bfdc-f5558df04093"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:53:33.695355 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.695262    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.695355 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.695322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "29cb07da-1c23-428b-bfdc-f5558df04093" (UID: "29cb07da-1c23-428b-bfdc-f5558df04093"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:33.697072 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.697046    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29cb07da-1c23-428b-bfdc-f5558df04093-kube-api-access-hzx8s" (OuterVolumeSpecName: "kube-api-access-hzx8s") pod "29cb07da-1c23-428b-bfdc-f5558df04093" (UID: "29cb07da-1c23-428b-bfdc-f5558df04093"). InnerVolumeSpecName "kube-api-access-hzx8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:33.697218 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.697193    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-dshm" (OuterVolumeSpecName: "dshm") pod "29cb07da-1c23-428b-bfdc-f5558df04093" (UID: "29cb07da-1c23-428b-bfdc-f5558df04093"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:33.697270 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.697242    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29cb07da-1c23-428b-bfdc-f5558df04093-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29cb07da-1c23-428b-bfdc-f5558df04093" (UID: "29cb07da-1c23-428b-bfdc-f5558df04093"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:53:33.753807 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.753740    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29cb07da-1c23-428b-bfdc-f5558df04093" (UID: "29cb07da-1c23-428b-bfdc-f5558df04093"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:53:33.796274 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.796246    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.796274 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.796270    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/29cb07da-1c23-428b-bfdc-f5558df04093-cabundle-cert\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.796447 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.796286    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.796447 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.796297    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29cb07da-1c23-428b-bfdc-f5558df04093-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.796447 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.796307    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.796447 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.796318    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29cb07da-1c23-428b-bfdc-f5558df04093-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:33.796447 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:33.796331    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzx8s\" (UniqueName: \"kubernetes.io/projected/29cb07da-1c23-428b-bfdc-f5558df04093-kube-api-access-hzx8s\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:34.144807 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.144772    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"
Jul 28 17:53:34.145307 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.144795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd" event={"ID":"348018a9-0918-47d5-b0f6-8d512138692c","Type":"ContainerDied","Data":"8287a6f07fa446aba927198f9cb7cee62d20ac57a73c61fd0a5cab96c58e8cb2"}
Jul 28 17:53:34.145307 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.144897    2579 scope.go:117] "RemoveContainer" containerID="363554868bfcc4c4a54849514976528b0869fdf9b98f0e32c0fa913551fc5c69"
Jul 28 17:53:34.146579 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.146545    2579 generic.go:356] "Generic (PLEG): container finished" podID="29cb07da-1c23-428b-bfdc-f5558df04093" containerID="e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85" exitCode=0
Jul 28 17:53:34.146689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.146598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" event={"ID":"29cb07da-1c23-428b-bfdc-f5558df04093","Type":"ContainerDied","Data":"e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85"}
Jul 28 17:53:34.146689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.146608    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"
Jul 28 17:53:34.146689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.146631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4" event={"ID":"29cb07da-1c23-428b-bfdc-f5558df04093","Type":"ContainerDied","Data":"d684fba86f052444b6a5c809f243c0745d1f22cb6b1d27703e1be1010051f06c"}
Jul 28 17:53:34.155347 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.155329    2579 scope.go:117] "RemoveContainer" containerID="e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85"
Jul 28 17:53:34.163687 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.163670    2579 scope.go:117] "RemoveContainer" containerID="6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431"
Jul 28 17:53:34.168351 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.168330    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"]
Jul 28 17:53:34.172217 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.172200    2579 scope.go:117] "RemoveContainer" containerID="e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85"
Jul 28 17:53:34.172460 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:53:34.172439    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85\": container with ID starting with e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85 not found: ID does not exist" containerID="e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85"
Jul 28 17:53:34.172536 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.172467    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85"} err="failed to get container status \"e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85\": rpc error: code = NotFound desc = could not find container \"e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85\": container with ID starting with e48b89ceb440cc7d70a47e9d148de1a6c3fdc8cc108b862f6ac998866e9bcb85 not found: ID does not exist"
Jul 28 17:53:34.172536 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.172483    2579 scope.go:117] "RemoveContainer" containerID="6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431"
Jul 28 17:53:34.172728 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:53:34.172708    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431\": container with ID starting with 6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431 not found: ID does not exist" containerID="6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431"
Jul 28 17:53:34.172779 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.172742    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431"} err="failed to get container status \"6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431\": rpc error: code = NotFound desc = could not find container \"6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431\": container with ID starting with 6cf0fa95d57a183d95ec81fd68b4ef337d3bafbe45beb2136471d0e373e01431 not found: ID does not exist"
Jul 28 17:53:34.172779 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.172714    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schejxbnd"]
Jul 28 17:53:34.183774 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.183752    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"]
Jul 28 17:53:34.187538 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.187517    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-767cc59c4b-dktt4"]
Jul 28 17:53:34.320629 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.320597    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" path="/var/lib/kubelet/pods/29cb07da-1c23-428b-bfdc-f5558df04093/volumes"
Jul 28 17:53:34.320990 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:34.320977    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="348018a9-0918-47d5-b0f6-8d512138692c" path="/var/lib/kubelet/pods/348018a9-0918-47d5-b0f6-8d512138692c/volumes"
Jul 28 17:53:40.720493 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.720458    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"]
Jul 28 17:53:40.720869 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.720860    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main"
Jul 28 17:53:40.720912 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.720877    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main"
Jul 28 17:53:40.720912 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.720904    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="348018a9-0918-47d5-b0f6-8d512138692c" containerName="main"
Jul 28 17:53:40.720912 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.720910    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="348018a9-0918-47d5-b0f6-8d512138692c" containerName="main"
Jul 28 17:53:40.721008 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.720917    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="storage-initializer"
Jul 28 17:53:40.721008 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.720924    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="storage-initializer"
Jul 28 17:53:40.721008 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.720983    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="348018a9-0918-47d5-b0f6-8d512138692c" containerName="main"
Jul 28 17:53:40.721008 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.720990    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="29cb07da-1c23-428b-bfdc-f5558df04093" containerName="main"
Jul 28 17:53:40.725009 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.724990    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:40.727854 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.727833    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 17:53:40.727984 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.727968    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-gsblg\""
Jul 28 17:53:40.728913 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.728896    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 17:53:40.735452 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.735431    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"]
Jul 28 17:53:40.855933 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.855907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5fkh\" (UniqueName: \"kubernetes.io/projected/f8ce3291-7888-4b3e-8860-25d9a55d883d-kube-api-access-m5fkh\") pod \"e2e-pvc-model-download-6vpgw\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:40.957196 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.957163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5fkh\" (UniqueName: \"kubernetes.io/projected/f8ce3291-7888-4b3e-8860-25d9a55d883d-kube-api-access-m5fkh\") pod \"e2e-pvc-model-download-6vpgw\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:40.967370 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:40.967344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5fkh\" (UniqueName: \"kubernetes.io/projected/f8ce3291-7888-4b3e-8860-25d9a55d883d-kube-api-access-m5fkh\") pod \"e2e-pvc-model-download-6vpgw\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:44.485548 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:44.485509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"e2e-pvc-model-download-6vpgw\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:44.491474 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:44.491447    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"e2e-pvc-model-download-6vpgw\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:44.586226 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:44.586196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"e2e-pvc-model-download-6vpgw\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:44.586352 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:44.586319    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"e2e-pvc-model-download-6vpgw\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:44.589741 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:44.589720    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"e2e-pvc-model-download-6vpgw\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") DevicePath \"csi-62bf3a9659f296df95c0a4f71d80bf260a63c38f0e91db765cfa0d632db7e784\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:44.822043 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:44.822010    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"e2e-pvc-model-download-6vpgw\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/79d3096c7b656d31344507b5ebc272ab902871c62304a1781d702496097fdff6/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:44.835918 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:44.835892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"e2e-pvc-model-download-6vpgw\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") " pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:44.934494 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:44.934463    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:45.065369 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:45.065344    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"]
Jul 28 17:53:45.070976 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:53:45.070947    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8ce3291_7888_4b3e_8860_25d9a55d883d.slice/crio-4a99197b416c0d56438bccf8e9cd3b87593693843040f36cc69b72337af0a8f4 WatchSource:0}: Error finding container 4a99197b416c0d56438bccf8e9cd3b87593693843040f36cc69b72337af0a8f4: Status 404 returned error can't find the container with id 4a99197b416c0d56438bccf8e9cd3b87593693843040f36cc69b72337af0a8f4
Jul 28 17:53:45.203637 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:45.203602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw" event={"ID":"f8ce3291-7888-4b3e-8860-25d9a55d883d","Type":"ContainerStarted","Data":"a4f152ffcdd0dc58c9842cf0b53dd9cc52c13545d81d24f41327ca4b57712641"}
Jul 28 17:53:45.203812 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:45.203646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw" event={"ID":"f8ce3291-7888-4b3e-8860-25d9a55d883d","Type":"ContainerStarted","Data":"4a99197b416c0d56438bccf8e9cd3b87593693843040f36cc69b72337af0a8f4"}
Jul 28 17:53:45.224256 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:45.224204    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw" podStartSLOduration=8.224190366 podStartE2EDuration="8.224190366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:53:45.223810388 +0000 UTC m=+2613.453009510" watchObservedRunningTime="2026-07-28 17:53:45.224190366 +0000 UTC m=+2613.453389510"
Jul 28 17:53:52.242449 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:52.242418    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8ce3291-7888-4b3e-8860-25d9a55d883d" containerID="a4f152ffcdd0dc58c9842cf0b53dd9cc52c13545d81d24f41327ca4b57712641" exitCode=0
Jul 28 17:53:52.242890 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:52.242496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw" event={"ID":"f8ce3291-7888-4b3e-8860-25d9a55d883d","Type":"ContainerDied","Data":"a4f152ffcdd0dc58c9842cf0b53dd9cc52c13545d81d24f41327ca4b57712641"}
Jul 28 17:53:53.388068 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:53.388046    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:53.472960 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:53.472931    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5fkh\" (UniqueName: \"kubernetes.io/projected/f8ce3291-7888-4b3e-8860-25d9a55d883d-kube-api-access-m5fkh\") pod \"f8ce3291-7888-4b3e-8860-25d9a55d883d\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") "
Jul 28 17:53:53.473130 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:53.473051    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"f8ce3291-7888-4b3e-8860-25d9a55d883d\" (UID: \"f8ce3291-7888-4b3e-8860-25d9a55d883d\") "
Jul 28 17:53:53.475293 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:53.475261    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8ce3291-7888-4b3e-8860-25d9a55d883d-kube-api-access-m5fkh" (OuterVolumeSpecName: "kube-api-access-m5fkh") pod "f8ce3291-7888-4b3e-8860-25d9a55d883d" (UID: "f8ce3291-7888-4b3e-8860-25d9a55d883d"). InnerVolumeSpecName "kube-api-access-m5fkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:53:53.477127 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:53.477103    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1" (OuterVolumeSpecName: "model-storage") pod "f8ce3291-7888-4b3e-8860-25d9a55d883d" (UID: "f8ce3291-7888-4b3e-8860-25d9a55d883d"). InnerVolumeSpecName "pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 17:53:53.574634 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:53.574598    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") on node \"ip-10-0-136-111.ec2.internal\" "
Jul 28 17:53:53.574634 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:53.574634    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5fkh\" (UniqueName: \"kubernetes.io/projected/f8ce3291-7888-4b3e-8860-25d9a55d883d-kube-api-access-m5fkh\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:53:53.690012 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:53.689990    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1") on node "ip-10-0-136-111.ec2.internal"
Jul 28 17:53:53.776472 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:53.776439    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"csi-62bf3a9659f296df95c0a4f71d80bf260a63c38f0e91db765cfa0d632db7e784\""
Jul 28 17:53:54.253552 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:54.253516    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"
Jul 28 17:53:54.253742 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:54.253564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw" event={"ID":"f8ce3291-7888-4b3e-8860-25d9a55d883d","Type":"ContainerDied","Data":"4a99197b416c0d56438bccf8e9cd3b87593693843040f36cc69b72337af0a8f4"}
Jul 28 17:53:54.253742 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:53:54.253602    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a99197b416c0d56438bccf8e9cd3b87593693843040f36cc69b72337af0a8f4"
Jul 28 17:54:10.351711 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.351669    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"]
Jul 28 17:54:10.352266 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.352246    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8ce3291-7888-4b3e-8860-25d9a55d883d" containerName="storage-initializer"
Jul 28 17:54:10.352343 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.352269    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8ce3291-7888-4b3e-8860-25d9a55d883d" containerName="storage-initializer"
Jul 28 17:54:10.352416 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.352403    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8ce3291-7888-4b3e-8860-25d9a55d883d" containerName="storage-initializer"
Jul 28 17:54:10.354700 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.354680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.357694 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.357660    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 28 17:54:10.357694 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.357693    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 28 17:54:10.357854 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.357691    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-gsblg\""
Jul 28 17:54:10.357854 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.357779    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 28 17:54:10.366770 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.366745    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"]
Jul 28 17:54:10.413855 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.413821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.414047 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.413885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.414047 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.413919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.414047 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.413959    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.414047 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.414022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.515008 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.514964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.515008 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.515009    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.515285 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.515047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.515285 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.515101    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstc2\" (UniqueName: \"kubernetes.io/projected/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-kube-api-access-xstc2\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.515285 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.515241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.515472 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.515299    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.515472 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.515445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.515588 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.515566    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.515652 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.515634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.517335 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.517313    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.517603 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.517585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.614098 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.613994    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"]
Jul 28 17:54:10.616811 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.616782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstc2\" (UniqueName: \"kubernetes.io/projected/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-kube-api-access-xstc2\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.617551 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.617532    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:54:10.620263 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.620237    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-9skvv\""
Jul 28 17:54:10.629931 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.629894    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"]
Jul 28 17:54:10.630666 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.630628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstc2\" (UniqueName: \"kubernetes.io/projected/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-kube-api-access-xstc2\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:10.718002 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.717958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbnsx\" (UniqueName: \"kubernetes.io/projected/4e04a0a4-f243-48c8-96a9-75b229387334-kube-api-access-rbnsx\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw\" (UID: \"4e04a0a4-f243-48c8-96a9-75b229387334\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:54:10.718002 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.718008    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e04a0a4-f243-48c8-96a9-75b229387334-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw\" (UID: \"4e04a0a4-f243-48c8-96a9-75b229387334\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:54:10.819340 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.819296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbnsx\" (UniqueName: \"kubernetes.io/projected/4e04a0a4-f243-48c8-96a9-75b229387334-kube-api-access-rbnsx\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw\" (UID: \"4e04a0a4-f243-48c8-96a9-75b229387334\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:54:10.819340 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.819351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e04a0a4-f243-48c8-96a9-75b229387334-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw\" (UID: \"4e04a0a4-f243-48c8-96a9-75b229387334\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:54:10.821862 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.821835    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e04a0a4-f243-48c8-96a9-75b229387334-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw\" (UID: \"4e04a0a4-f243-48c8-96a9-75b229387334\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:54:10.827792 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.827764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbnsx\" (UniqueName: \"kubernetes.io/projected/4e04a0a4-f243-48c8-96a9-75b229387334-kube-api-access-rbnsx\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw\" (UID: \"4e04a0a4-f243-48c8-96a9-75b229387334\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:54:10.931647 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:10.931566    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:54:11.077137 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:11.077109    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"]
Jul 28 17:54:11.078928 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:54:11.078903    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e04a0a4_f243_48c8_96a9_75b229387334.slice/crio-da427f66efd384655ceb79e12294a94c2a4bc3cd987a84feed625dd883cff718 WatchSource:0}: Error finding container da427f66efd384655ceb79e12294a94c2a4bc3cd987a84feed625dd883cff718: Status 404 returned error can't find the container with id da427f66efd384655ceb79e12294a94c2a4bc3cd987a84feed625dd883cff718
Jul 28 17:54:11.336830 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:11.336787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw" event={"ID":"4e04a0a4-f243-48c8-96a9-75b229387334","Type":"ContainerStarted","Data":"f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc"}
Jul 28 17:54:11.337008 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:11.336849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw" event={"ID":"4e04a0a4-f243-48c8-96a9-75b229387334","Type":"ContainerStarted","Data":"da427f66efd384655ceb79e12294a94c2a4bc3cd987a84feed625dd883cff718"}
Jul 28 17:54:11.337008 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:11.336870    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:54:11.357593 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:11.357543    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw" podStartSLOduration=1.35752826 podStartE2EDuration="1.35752826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:11.356462166 +0000 UTC m=+2639.585661263" watchObservedRunningTime="2026-07-28 17:54:11.35752826 +0000 UTC m=+2639.586727356"
Jul 28 17:54:13.441972 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:13.441936    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:13.449494 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:13.449465    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") device path: \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:13.543126 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:13.543092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:13.543285 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:13.543163    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") DevicePath \"\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:13.546462 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:13.546442    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") DevicePath \"csi-62bf3a9659f296df95c0a4f71d80bf260a63c38f0e91db765cfa0d632db7e784\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:13.635658 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:13.635628    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/79d3096c7b656d31344507b5ebc272ab902871c62304a1781d702496097fdff6/globalmount\"" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:13.652234 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:13.652207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:13.665031 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:13.665003    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:13.805900 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:13.805851    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"]
Jul 28 17:54:13.808162 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:54:13.808134    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69d5fae_83ba_41b5_b09b_18ae45b8aa6e.slice/crio-5f85210ab98605e6f783352d98a7a8c40befcdbc2bd51b311565414aae3145db WatchSource:0}: Error finding container 5f85210ab98605e6f783352d98a7a8c40befcdbc2bd51b311565414aae3145db: Status 404 returned error can't find the container with id 5f85210ab98605e6f783352d98a7a8c40befcdbc2bd51b311565414aae3145db
Jul 28 17:54:14.352671 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:14.352630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" event={"ID":"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e","Type":"ContainerStarted","Data":"1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d"}
Jul 28 17:54:14.352860 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:14.352699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" event={"ID":"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e","Type":"ContainerStarted","Data":"5f85210ab98605e6f783352d98a7a8c40befcdbc2bd51b311565414aae3145db"}
Jul 28 17:54:14.378048 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:14.377973    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podStartSLOduration=4.377957961 podStartE2EDuration="4.377957961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:54:14.375303574 +0000 UTC m=+2642.604502670" watchObservedRunningTime="2026-07-28 17:54:14.377957961 +0000 UTC m=+2642.607157057"
Jul 28 17:54:14.665421 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:14.665332    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:23.665594 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:23.665557    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:54:23.667170 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:23.667141    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:54:33.666429 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:33.666382    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:54:42.343474 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:42.343443    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:54:43.665753 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:43.665697    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:54:53.665534 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:54:53.665490    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:55:03.666255 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:55:03.666200    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:55:12.427159 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:55:12.427130    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:55:12.435622 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:55:12.435596    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 17:55:13.665576 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:55:13.665534    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:55:23.665836 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:55:23.665784    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:55:33.665497 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:55:33.665456    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:55:43.666479 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:55:43.666432    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:55:53.665990 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:55:53.665933    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:56:03.675592 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:03.675558    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:56:03.683303 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:03.683279    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:56:09.006820 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:09.006784    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"]
Jul 28 17:56:09.019859 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:09.019827    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"]
Jul 28 17:56:12.343198 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:12.343163    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw" podUID="4e04a0a4-f243-48c8-96a9-75b229387334" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:56:22.343304 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:22.343267    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw" podUID="4e04a0a4-f243-48c8-96a9-75b229387334" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 17:56:24.029409 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.029373    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" containerID="cri-o://1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d" gracePeriod=45
Jul 28 17:56:24.039785 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.039730    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw" podUID="4e04a0a4-f243-48c8-96a9-75b229387334" containerName="main" containerID="cri-o://f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc" gracePeriod=45
Jul 28 17:56:24.296887 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.296829    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:56:24.352239 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.352207    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbnsx\" (UniqueName: \"kubernetes.io/projected/4e04a0a4-f243-48c8-96a9-75b229387334-kube-api-access-rbnsx\") pod \"4e04a0a4-f243-48c8-96a9-75b229387334\" (UID: \"4e04a0a4-f243-48c8-96a9-75b229387334\") "
Jul 28 17:56:24.352416 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.352258    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e04a0a4-f243-48c8-96a9-75b229387334-tls-certs\") pod \"4e04a0a4-f243-48c8-96a9-75b229387334\" (UID: \"4e04a0a4-f243-48c8-96a9-75b229387334\") "
Jul 28 17:56:24.354343 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.354311    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e04a0a4-f243-48c8-96a9-75b229387334-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e04a0a4-f243-48c8-96a9-75b229387334" (UID: "4e04a0a4-f243-48c8-96a9-75b229387334"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:56:24.354456 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.354379    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e04a0a4-f243-48c8-96a9-75b229387334-kube-api-access-rbnsx" (OuterVolumeSpecName: "kube-api-access-rbnsx") pod "4e04a0a4-f243-48c8-96a9-75b229387334" (UID: "4e04a0a4-f243-48c8-96a9-75b229387334"). InnerVolumeSpecName "kube-api-access-rbnsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:56:24.453674 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.453636    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbnsx\" (UniqueName: \"kubernetes.io/projected/4e04a0a4-f243-48c8-96a9-75b229387334-kube-api-access-rbnsx\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:56:24.453674 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.453669    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e04a0a4-f243-48c8-96a9-75b229387334-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:56:24.677135 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.677096    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": dial tcp 10.132.0.67:8000: connect: connection refused"
Jul 28 17:56:24.772615 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.772590    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:56:24.857608 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.857508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tls-certs\") pod \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") "
Jul 28 17:56:24.857793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.857643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") pod \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") "
Jul 28 17:56:24.857793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.857669    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-model-cache\") pod \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") "
Jul 28 17:56:24.857793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.857708    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xstc2\" (UniqueName: \"kubernetes.io/projected/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-kube-api-access-xstc2\") pod \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") "
Jul 28 17:56:24.857793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.857759    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-dshm\") pod \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") "
Jul 28 17:56:24.857793 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.857781    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tmp-dir\") pod \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") "
Jul 28 17:56:24.858126 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.857809    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-home\") pod \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\" (UID: \"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e\") "
Jul 28 17:56:24.858126 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.857930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" (UID: "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:24.858315 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.858131    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:56:24.858640 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.858600    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-home" (OuterVolumeSpecName: "home") pod "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" (UID: "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:24.860028 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.860003    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" (UID: "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:56:24.860028 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.860004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-kube-api-access-xstc2" (OuterVolumeSpecName: "kube-api-access-xstc2") pod "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" (UID: "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e"). InnerVolumeSpecName "kube-api-access-xstc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:56:24.860546 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.860520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-dshm" (OuterVolumeSpecName: "dshm") pod "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" (UID: "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:24.860945 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.860924    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1" (OuterVolumeSpecName: "kserve-pvc-source") pod "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" (UID: "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e"). InnerVolumeSpecName "pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 17:56:24.868689 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.868667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" (UID: "f69d5fae-83ba-41b5-b09b-18ae45b8aa6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:56:24.959519 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.959481    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:56:24.959687 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.959533    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") on node \"ip-10-0-136-111.ec2.internal\" "
Jul 28 17:56:24.959687 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.959551    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstc2\" (UniqueName: \"kubernetes.io/projected/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-kube-api-access-xstc2\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:56:24.959687 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.959563    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:56:24.959687 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.959575    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:56:24.959687 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.959587    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:56:24.988291 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:24.988269    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1") on node "ip-10-0-136-111.ec2.internal"
Jul 28 17:56:25.025290 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.025259    2579 generic.go:356] "Generic (PLEG): container finished" podID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerID="1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d" exitCode=0
Jul 28 17:56:25.025437 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.025302    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"
Jul 28 17:56:25.025437 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.025344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" event={"ID":"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e","Type":"ContainerDied","Data":"1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d"}
Jul 28 17:56:25.025437 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.025411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn" event={"ID":"f69d5fae-83ba-41b5-b09b-18ae45b8aa6e","Type":"ContainerDied","Data":"5f85210ab98605e6f783352d98a7a8c40befcdbc2bd51b311565414aae3145db"}
Jul 28 17:56:25.025584 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.025446    2579 scope.go:117] "RemoveContainer" containerID="1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d"
Jul 28 17:56:25.026674 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.026607    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e04a0a4-f243-48c8-96a9-75b229387334" containerID="f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc" exitCode=0
Jul 28 17:56:25.026674 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.026663    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"
Jul 28 17:56:25.026821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.026693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw" event={"ID":"4e04a0a4-f243-48c8-96a9-75b229387334","Type":"ContainerDied","Data":"f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc"}
Jul 28 17:56:25.026821 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.026745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw" event={"ID":"4e04a0a4-f243-48c8-96a9-75b229387334","Type":"ContainerDied","Data":"da427f66efd384655ceb79e12294a94c2a4bc3cd987a84feed625dd883cff718"}
Jul 28 17:56:25.035978 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.035691    2579 scope.go:117] "RemoveContainer" containerID="1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d"
Jul 28 17:56:25.036271 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:56:25.036005    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d\": container with ID starting with 1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d not found: ID does not exist" containerID="1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d"
Jul 28 17:56:25.036271 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.036027    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d"} err="failed to get container status \"1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d\": rpc error: code = NotFound desc = could not find container \"1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d\": container with ID starting with 1c205a1dde66fe80a9b164ab45804810294039c2559349887a1d074f4ade907d not found: ID does not exist"
Jul 28 17:56:25.036271 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.036043    2579 scope.go:117] "RemoveContainer" containerID="f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc"
Jul 28 17:56:25.044199 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.044108    2579 scope.go:117] "RemoveContainer" containerID="f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc"
Jul 28 17:56:25.044375 ip-10-0-136-111 kubenswrapper[2579]: E0728 17:56:25.044358    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc\": container with ID starting with f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc not found: ID does not exist" containerID="f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc"
Jul 28 17:56:25.044423 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.044383    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc"} err="failed to get container status \"f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc\": rpc error: code = NotFound desc = could not find container \"f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc\": container with ID starting with f028c4b88db0a1ff1d59a9e91ddd249b8fab63fad9e421d1914d690bb887d7dc not found: ID does not exist"
Jul 28 17:56:25.052891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.052871    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"]
Jul 28 17:56:25.058065 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.058043    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc-model-pvc-router-manage-59b9d263-kserve-6c6f7bfcfcjppmn"]
Jul 28 17:56:25.060971 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.060952    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-d19ce3da-601f-41bb-9f5c-92dec1a5f7cb\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0cd104526988987a1\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"csi-62bf3a9659f296df95c0a4f71d80bf260a63c38f0e91db765cfa0d632db7e784\""
Jul 28 17:56:25.067625 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.067603    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"]
Jul 28 17:56:25.072646 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:25.072628    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-schew6qhw"]
Jul 28 17:56:26.319448 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:26.319417    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e04a0a4-f243-48c8-96a9-75b229387334" path="/var/lib/kubelet/pods/4e04a0a4-f243-48c8-96a9-75b229387334/volumes"
Jul 28 17:56:26.319843 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:26.319830    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" path="/var/lib/kubelet/pods/f69d5fae-83ba-41b5-b09b-18ae45b8aa6e/volumes"
Jul 28 17:56:31.970566 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.970532    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"]
Jul 28 17:56:31.970961 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.970888    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e04a0a4-f243-48c8-96a9-75b229387334" containerName="main"
Jul 28 17:56:31.970961 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.970899    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e04a0a4-f243-48c8-96a9-75b229387334" containerName="main"
Jul 28 17:56:31.970961 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.970921    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main"
Jul 28 17:56:31.970961 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.970927    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main"
Jul 28 17:56:31.971118 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.970981    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f69d5fae-83ba-41b5-b09b-18ae45b8aa6e" containerName="main"
Jul 28 17:56:31.971118 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.970990    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e04a0a4-f243-48c8-96a9-75b229387334" containerName="main"
Jul 28 17:56:31.975716 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.975697    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:31.978356 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.978331    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 17:56:31.978552 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.978538    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 17:56:31.979639 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.979525    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-schcf\""
Jul 28 17:56:31.982124 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:31.982103    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"]
Jul 28 17:56:32.017775 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:32.017746    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bllp\" (UniqueName: \"kubernetes.io/projected/e0265ec5-01a4-40e9-9cab-64832ae67a91-kube-api-access-2bllp\") pod \"e2e-pvc-model-download-rv8z8\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:32.118788 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:32.118761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bllp\" (UniqueName: \"kubernetes.io/projected/e0265ec5-01a4-40e9-9cab-64832ae67a91-kube-api-access-2bllp\") pod \"e2e-pvc-model-download-rv8z8\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:32.127247 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:32.127223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bllp\" (UniqueName: \"kubernetes.io/projected/e0265ec5-01a4-40e9-9cab-64832ae67a91-kube-api-access-2bllp\") pod \"e2e-pvc-model-download-rv8z8\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:33.677940 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:33.677907    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"]
Jul 28 17:56:33.681406 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:33.681380    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/e2e-pvc-model-download-6vpgw"]
Jul 28 17:56:34.320462 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:34.320386    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8ce3291-7888-4b3e-8860-25d9a55d883d" path="/var/lib/kubelet/pods/f8ce3291-7888-4b3e-8860-25d9a55d883d/volumes"
Jul 28 17:56:37.263088 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:37.263038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"e2e-pvc-model-download-rv8z8\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:37.269391 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:37.269361    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"e2e-pvc-model-download-rv8z8\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:37.364399 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:37.364372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"e2e-pvc-model-download-rv8z8\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:37.364530 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:37.364513    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"e2e-pvc-model-download-rv8z8\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:37.367676 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:37.367657    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"e2e-pvc-model-download-rv8z8\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") DevicePath \"csi-3fb48e1459c0c2d3ec4c0ecd555366d6d447cf81bdb9745be39d0f42c00ce130\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:37.607240 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:37.607204    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"e2e-pvc-model-download-rv8z8\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/e0f8b91196490002db60df69f72ca19008f0180a47f25d0ef2c2da7ac9fce7b5/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:37.621169 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:37.621147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"e2e-pvc-model-download-rv8z8\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") " pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:37.685931 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:37.685905    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:37.813292 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:37.813265    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"]
Jul 28 17:56:37.815927 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:56:37.815899    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0265ec5_01a4_40e9_9cab_64832ae67a91.slice/crio-cbc923750d2db42bcfc99e568ca9330912bebf6c4b54cb6894875987455c0eda WatchSource:0}: Error finding container cbc923750d2db42bcfc99e568ca9330912bebf6c4b54cb6894875987455c0eda: Status 404 returned error can't find the container with id cbc923750d2db42bcfc99e568ca9330912bebf6c4b54cb6894875987455c0eda
Jul 28 17:56:38.098200 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:38.098109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8" event={"ID":"e0265ec5-01a4-40e9-9cab-64832ae67a91","Type":"ContainerStarted","Data":"046a7dee40a106a8e0962838f282e47ed582b73599b2005b74f23215ace833b5"}
Jul 28 17:56:38.098200 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:38.098157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8" event={"ID":"e0265ec5-01a4-40e9-9cab-64832ae67a91","Type":"ContainerStarted","Data":"cbc923750d2db42bcfc99e568ca9330912bebf6c4b54cb6894875987455c0eda"}
Jul 28 17:56:38.117758 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:38.117706    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8" podStartSLOduration=10.117685961 podStartE2EDuration="10.117685961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:56:38.115935057 +0000 UTC m=+2786.345134157" watchObservedRunningTime="2026-07-28 17:56:38.117685961 +0000 UTC m=+2786.346885074"
Jul 28 17:56:45.135587 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:45.135547    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0265ec5-01a4-40e9-9cab-64832ae67a91" containerID="046a7dee40a106a8e0962838f282e47ed582b73599b2005b74f23215ace833b5" exitCode=0
Jul 28 17:56:45.136196 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:45.135622    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8" event={"ID":"e0265ec5-01a4-40e9-9cab-64832ae67a91","Type":"ContainerDied","Data":"046a7dee40a106a8e0962838f282e47ed582b73599b2005b74f23215ace833b5"}
Jul 28 17:56:46.292347 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:46.292321    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:46.349835 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:46.349803    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bllp\" (UniqueName: \"kubernetes.io/projected/e0265ec5-01a4-40e9-9cab-64832ae67a91-kube-api-access-2bllp\") pod \"e0265ec5-01a4-40e9-9cab-64832ae67a91\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") "
Jul 28 17:56:46.349988 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:46.349917    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"e0265ec5-01a4-40e9-9cab-64832ae67a91\" (UID: \"e0265ec5-01a4-40e9-9cab-64832ae67a91\") "
Jul 28 17:56:46.352176 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:46.352138    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0265ec5-01a4-40e9-9cab-64832ae67a91-kube-api-access-2bllp" (OuterVolumeSpecName: "kube-api-access-2bllp") pod "e0265ec5-01a4-40e9-9cab-64832ae67a91" (UID: "e0265ec5-01a4-40e9-9cab-64832ae67a91"). InnerVolumeSpecName "kube-api-access-2bllp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:56:46.352577 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:46.352557    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a" (OuterVolumeSpecName: "model-storage") pod "e0265ec5-01a4-40e9-9cab-64832ae67a91" (UID: "e0265ec5-01a4-40e9-9cab-64832ae67a91"). InnerVolumeSpecName "pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 17:56:46.451743 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:46.451633    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bllp\" (UniqueName: \"kubernetes.io/projected/e0265ec5-01a4-40e9-9cab-64832ae67a91-kube-api-access-2bllp\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 17:56:46.451743 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:46.451704    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") on node \"ip-10-0-136-111.ec2.internal\" "
Jul 28 17:56:46.567933 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:46.567912    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a") on node "ip-10-0-136-111.ec2.internal"
Jul 28 17:56:46.653376 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:46.653342    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"csi-3fb48e1459c0c2d3ec4c0ecd555366d6d447cf81bdb9745be39d0f42c00ce130\""
Jul 28 17:56:47.146678 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:47.146639    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"
Jul 28 17:56:47.146678 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:47.146659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8" event={"ID":"e0265ec5-01a4-40e9-9cab-64832ae67a91","Type":"ContainerDied","Data":"cbc923750d2db42bcfc99e568ca9330912bebf6c4b54cb6894875987455c0eda"}
Jul 28 17:56:47.146896 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:47.146694    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbc923750d2db42bcfc99e568ca9330912bebf6c4b54cb6894875987455c0eda"
Jul 28 17:56:56.598179 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.598142    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"]
Jul 28 17:56:56.598749 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.598732    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0265ec5-01a4-40e9-9cab-64832ae67a91" containerName="storage-initializer"
Jul 28 17:56:56.598797 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.598753    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0265ec5-01a4-40e9-9cab-64832ae67a91" containerName="storage-initializer"
Jul 28 17:56:56.598865 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.598852    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0265ec5-01a4-40e9-9cab-64832ae67a91" containerName="storage-initializer"
Jul 28 17:56:56.602446 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.602426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.605782 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.605759    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 28 17:56:56.605894 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.605807    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 28 17:56:56.605962 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.605949    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 28 17:56:56.606901 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.606871    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-wpdjr\""
Jul 28 17:56:56.607027 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.606939    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-schcf\""
Jul 28 17:56:56.616022 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.615997    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"]
Jul 28 17:56:56.619634 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.619616    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"]
Jul 28 17:56:56.619751 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.619720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.630680 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.630653    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"]
Jul 28 17:56:56.743367 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.743367 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.743600 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.743600 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2f7\" (UniqueName: \"kubernetes.io/projected/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-kube-api-access-pz2f7\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.743600 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.743600 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743533    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfz9\" (UniqueName: \"kubernetes.io/projected/a4ec3e8c-6f13-41ce-96b5-677757a0d026-kube-api-access-mlfz9\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.743600 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.743600 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.743896 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743616    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.743896 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.743896 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.743896 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.743770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.845120 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.845120 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845118    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.845384 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845150    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.845384 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2f7\" (UniqueName: \"kubernetes.io/projected/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-kube-api-access-pz2f7\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.845384 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.845384 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfz9\" (UniqueName: \"kubernetes.io/projected/a4ec3e8c-6f13-41ce-96b5-677757a0d026-kube-api-access-mlfz9\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.845384 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845232    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.845384 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.845384 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.845384 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.845384 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.845891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.845891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.845891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.845891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.845891 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.845813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.846310 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.846291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.846385 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.846346    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.847876 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.847850    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.847984 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.847878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.847984 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.847941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:56:56.848246 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.848204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.861706 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.861680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2f7\" (UniqueName: \"kubernetes.io/projected/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-kube-api-access-pz2f7\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:56:56.861797 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:56:56.861739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfz9\" (UniqueName: \"kubernetes.io/projected/a4ec3e8c-6f13-41ce-96b5-677757a0d026-kube-api-access-mlfz9\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:57:00.376720 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.376681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:57:00.385429 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.385398    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") device path: \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:57:00.478094 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.478050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:57:00.478274 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.478191    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") DevicePath \"\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:57:00.481257 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.481235    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") DevicePath \"csi-3fb48e1459c0c2d3ec4c0ecd555366d6d447cf81bdb9745be39d0f42c00ce130\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:57:00.566233 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.566194    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/e0f8b91196490002db60df69f72ca19008f0180a47f25d0ef2c2da7ac9fce7b5/globalmount\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:57:00.583133 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.583070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:57:00.680912 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.680816    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:57:00.681124 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.680964    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") DevicePath \"csi-3fb48e1459c0c2d3ec4c0ecd555366d6d447cf81bdb9745be39d0f42c00ce130\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:57:00.684360 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.684341    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") DevicePath \"csi-3fb48e1459c0c2d3ec4c0ecd555366d6d447cf81bdb9745be39d0f42c00ce130\"" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:57:00.696286 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.696261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:57:00.813552 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.813516    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:57:00.830158 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.830124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:57:00.995257 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.971506    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"]
Jul 28 17:57:00.995257 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:00.987486    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"]
Jul 28 17:57:00.997382 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:57:00.997336    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ec3e8c_6f13_41ce_96b5_677757a0d026.slice/crio-b19a06d87e00eb1ae5496439845cd7fdb4eda67f3b8528fe916d2e1d0030cb8c WatchSource:0}: Error finding container b19a06d87e00eb1ae5496439845cd7fdb4eda67f3b8528fe916d2e1d0030cb8c: Status 404 returned error can't find the container with id b19a06d87e00eb1ae5496439845cd7fdb4eda67f3b8528fe916d2e1d0030cb8c
Jul 28 17:57:00.999433 ip-10-0-136-111 kubenswrapper[2579]: W0728 17:57:00.999409    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b33e9ac_31d4_4954_8a07_80e7f4e41bcb.slice/crio-964114a5e20ee7062759c7f9f95e392e31f78c17e99d90fa3912ea45c1344f91 WatchSource:0}: Error finding container 964114a5e20ee7062759c7f9f95e392e31f78c17e99d90fa3912ea45c1344f91: Status 404 returned error can't find the container with id 964114a5e20ee7062759c7f9f95e392e31f78c17e99d90fa3912ea45c1344f91
Jul 28 17:57:01.214394 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:01.214350    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" event={"ID":"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb","Type":"ContainerStarted","Data":"e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e"}
Jul 28 17:57:01.214844 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:01.214414    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" event={"ID":"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb","Type":"ContainerStarted","Data":"964114a5e20ee7062759c7f9f95e392e31f78c17e99d90fa3912ea45c1344f91"}
Jul 28 17:57:01.216151 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:01.216119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" event={"ID":"a4ec3e8c-6f13-41ce-96b5-677757a0d026","Type":"ContainerStarted","Data":"0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c"}
Jul 28 17:57:01.216258 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:01.216169    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:57:01.216258 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:01.216189    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" event={"ID":"a4ec3e8c-6f13-41ce-96b5-677757a0d026","Type":"ContainerStarted","Data":"b19a06d87e00eb1ae5496439845cd7fdb4eda67f3b8528fe916d2e1d0030cb8c"}
Jul 28 17:57:01.233981 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:01.233937    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podStartSLOduration=5.23391934 podStartE2EDuration="5.23391934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:01.232686548 +0000 UTC m=+2809.461885657" watchObservedRunningTime="2026-07-28 17:57:01.23391934 +0000 UTC m=+2809.463118435"
Jul 28 17:57:02.223837 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:02.223804    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" event={"ID":"a4ec3e8c-6f13-41ce-96b5-677757a0d026","Type":"ContainerStarted","Data":"73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d"}
Jul 28 17:57:02.247005 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:02.246923    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podStartSLOduration=6.246899916 podStartE2EDuration="6.246899916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:02.246159409 +0000 UTC m=+2810.475358509" watchObservedRunningTime="2026-07-28 17:57:02.246899916 +0000 UTC m=+2810.476099013"
Jul 28 17:57:10.814247 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:10.814206    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:57:10.814710 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:10.814307    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:57:10.815594 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:10.815562    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:57:10.830983 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:10.830951    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:57:10.831140 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:10.831024    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:57:10.832294 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:10.832262    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:57:20.814479 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:20.814430    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:57:20.831040 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:20.831000    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:57:20.832903 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:20.832671    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:57:30.814405 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:30.814353    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:57:30.830829 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:30.830783    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:57:40.814885 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:40.814836    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:57:40.830542 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:40.830488    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:57:50.814535 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:50.814481    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:57:50.830640 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:57:50.830598    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:58:00.814058 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:00.814002    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:58:00.830571 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:00.830518    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:58:10.814572 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:10.814458    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:58:10.830832 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:10.830779    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:58:20.814230 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:20.814172    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:58:20.831171 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:20.831130    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:58:30.814686 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:30.814632    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:58:30.831381 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:30.831328    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:58:40.814041 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:40.813983    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:58:40.830928 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:40.830885    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:58:50.814428 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:50.814367    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:58:50.830661 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:58:50.830611    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:59:00.814733 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:00.814661    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:59:00.830417 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:00.830377    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:59:10.814451 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:10.814402    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:59:10.831379 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:10.831327    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:59:20.814714 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:20.814665    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:59:20.831089 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:20.831036    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:59:30.814101 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:30.814034    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8001/health\": dial tcp 10.132.0.70:8001: connect: connection refused"
Jul 28 17:59:30.830777 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:30.830731    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": dial tcp 10.132.0.71:8000: connect: connection refused"
Jul 28 17:59:40.824991 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:40.824903    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:59:40.840260 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:40.840231    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 17:59:40.840440 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:40.840424    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:59:40.847854 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:40.847834    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 17:59:48.763571 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:48.763537    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"]
Jul 28 17:59:48.773392 ip-10-0-136-111 kubenswrapper[2579]: I0728 17:59:48.773363    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"]
Jul 28 18:00:03.794558 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:03.794494    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main" containerID="cri-o://73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d" gracePeriod=45
Jul 28 18:00:03.795834 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:03.795807    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main" containerID="cri-o://e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e" gracePeriod=45
Jul 28 18:00:04.605829 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.605798    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="llm-d-routing-sidecar" containerID="cri-o://0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c" gracePeriod=45
Jul 28 18:00:04.747172 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.747145    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 18:00:04.798748 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.798674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-model-cache\") pod \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") "
Jul 28 18:00:04.798748 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.798720    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz2f7\" (UniqueName: \"kubernetes.io/projected/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-kube-api-access-pz2f7\") pod \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") "
Jul 28 18:00:04.798748 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.798746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-home\") pod \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") "
Jul 28 18:00:04.799334 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.798831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-dshm\") pod \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") "
Jul 28 18:00:04.799334 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.798878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tmp-dir\") pod \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") "
Jul 28 18:00:04.799334 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.798949    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tls-certs\") pod \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") "
Jul 28 18:00:04.799334 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.798970    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" (UID: "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:04.799334 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.799056    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\" (UID: \"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb\") "
Jul 28 18:00:04.799555 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.799394    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:04.799555 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.799517    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-home" (OuterVolumeSpecName: "home") pod "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" (UID: "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:04.801365 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.801322    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-kube-api-access-pz2f7" (OuterVolumeSpecName: "kube-api-access-pz2f7") pod "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" (UID: "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb"). InnerVolumeSpecName "kube-api-access-pz2f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:04.801507 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.801430    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-dshm" (OuterVolumeSpecName: "dshm") pod "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" (UID: "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:04.801838 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.801811    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" (UID: "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:00:04.802600 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.802581    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a" (OuterVolumeSpecName: "kserve-pvc-source") pod "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" (UID: "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb"). InnerVolumeSpecName "pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:00:04.810998 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.810974    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" (UID: "8b33e9ac-31d4-4954-8a07-80e7f4e41bcb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:04.845168 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.845148    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 18:00:04.900305 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tmp-dir\") pod \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") "
Jul 28 18:00:04.900478 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900323    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-dshm\") pod \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") "
Jul 28 18:00:04.900478 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-model-cache\") pod \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") "
Jul 28 18:00:04.900478 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900391    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tls-certs\") pod \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") "
Jul 28 18:00:04.900644 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900556    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") pod \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") "
Jul 28 18:00:04.900644 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900629    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-home\") pod \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") "
Jul 28 18:00:04.900755 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900633    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-model-cache" (OuterVolumeSpecName: "model-cache") pod "a4ec3e8c-6f13-41ce-96b5-677757a0d026" (UID: "a4ec3e8c-6f13-41ce-96b5-677757a0d026"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:04.900755 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlfz9\" (UniqueName: \"kubernetes.io/projected/a4ec3e8c-6f13-41ce-96b5-677757a0d026-kube-api-access-mlfz9\") pod \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\" (UID: \"a4ec3e8c-6f13-41ce-96b5-677757a0d026\") "
Jul 28 18:00:04.900898 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900880    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz2f7\" (UniqueName: \"kubernetes.io/projected/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-kube-api-access-pz2f7\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:04.900955 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900899    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:04.900955 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900911    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:04.900955 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900924    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:04.900955 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900937    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:04.900955 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.900949    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:04.902519 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.902489    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-home" (OuterVolumeSpecName: "home") pod "a4ec3e8c-6f13-41ce-96b5-677757a0d026" (UID: "a4ec3e8c-6f13-41ce-96b5-677757a0d026"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:04.903119 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.903094    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-dshm" (OuterVolumeSpecName: "dshm") pod "a4ec3e8c-6f13-41ce-96b5-677757a0d026" (UID: "a4ec3e8c-6f13-41ce-96b5-677757a0d026"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:04.903234 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.903163    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4ec3e8c-6f13-41ce-96b5-677757a0d026-kube-api-access-mlfz9" (OuterVolumeSpecName: "kube-api-access-mlfz9") pod "a4ec3e8c-6f13-41ce-96b5-677757a0d026" (UID: "a4ec3e8c-6f13-41ce-96b5-677757a0d026"). InnerVolumeSpecName "kube-api-access-mlfz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:04.903504 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.903479    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a4ec3e8c-6f13-41ce-96b5-677757a0d026" (UID: "a4ec3e8c-6f13-41ce-96b5-677757a0d026"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:00:04.903934 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.903914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a" (OuterVolumeSpecName: "kserve-pvc-source") pod "a4ec3e8c-6f13-41ce-96b5-677757a0d026" (UID: "a4ec3e8c-6f13-41ce-96b5-677757a0d026"). InnerVolumeSpecName "pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:00:04.912654 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:04.912630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a4ec3e8c-6f13-41ce-96b5-677757a0d026" (UID: "a4ec3e8c-6f13-41ce-96b5-677757a0d026"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:00:05.001608 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.001573    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") on node \"ip-10-0-136-111.ec2.internal\" "
Jul 28 18:00:05.001608 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.001605    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:05.001791 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.001618    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlfz9\" (UniqueName: \"kubernetes.io/projected/a4ec3e8c-6f13-41ce-96b5-677757a0d026-kube-api-access-mlfz9\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:05.001791 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.001633    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:05.001791 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.001646    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a4ec3e8c-6f13-41ce-96b5-677757a0d026-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:05.001791 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.001657    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a4ec3e8c-6f13-41ce-96b5-677757a0d026-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:05.029439 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.029419    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a") on node "ip-10-0-136-111.ec2.internal"
Jul 28 18:00:05.102962 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.102929    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-65c8e9ca-444e-41f1-8163-52a2b066fe28\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0ebc8b5f73f14355a\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"csi-3fb48e1459c0c2d3ec4c0ecd555366d6d447cf81bdb9745be39d0f42c00ce130\""
Jul 28 18:00:05.205350 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.205318    2579 generic.go:356] "Generic (PLEG): container finished" podID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerID="e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e" exitCode=0
Jul 28 18:00:05.205533 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.205377    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"
Jul 28 18:00:05.205533 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.205402    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" event={"ID":"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb","Type":"ContainerDied","Data":"e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e"}
Jul 28 18:00:05.205533 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.205484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv" event={"ID":"8b33e9ac-31d4-4954-8a07-80e7f4e41bcb","Type":"ContainerDied","Data":"964114a5e20ee7062759c7f9f95e392e31f78c17e99d90fa3912ea45c1344f91"}
Jul 28 18:00:05.205533 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.205516    2579 scope.go:117] "RemoveContainer" containerID="e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e"
Jul 28 18:00:05.207008 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.206987    2579 generic.go:356] "Generic (PLEG): container finished" podID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerID="73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d" exitCode=0
Jul 28 18:00:05.207126 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.207010    2579 generic.go:356] "Generic (PLEG): container finished" podID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerID="0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c" exitCode=0
Jul 28 18:00:05.207126 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.207053    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"
Jul 28 18:00:05.207126 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.207073    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" event={"ID":"a4ec3e8c-6f13-41ce-96b5-677757a0d026","Type":"ContainerDied","Data":"73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d"}
Jul 28 18:00:05.207257 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.207138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" event={"ID":"a4ec3e8c-6f13-41ce-96b5-677757a0d026","Type":"ContainerDied","Data":"0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c"}
Jul 28 18:00:05.207257 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.207168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl" event={"ID":"a4ec3e8c-6f13-41ce-96b5-677757a0d026","Type":"ContainerDied","Data":"b19a06d87e00eb1ae5496439845cd7fdb4eda67f3b8528fe916d2e1d0030cb8c"}
Jul 28 18:00:05.215760 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.215743    2579 scope.go:117] "RemoveContainer" containerID="e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e"
Jul 28 18:00:05.216030 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:00:05.216009    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e\": container with ID starting with e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e not found: ID does not exist" containerID="e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e"
Jul 28 18:00:05.216125 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.216040    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e"} err="failed to get container status \"e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e\": rpc error: code = NotFound desc = could not find container \"e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e\": container with ID starting with e75b88b0e52bff9a4605ac11197ca855d764221e4236bf4d0ddeb446cee2671e not found: ID does not exist"
Jul 28 18:00:05.216125 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.216065    2579 scope.go:117] "RemoveContainer" containerID="73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d"
Jul 28 18:00:05.224123 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.224106    2579 scope.go:117] "RemoveContainer" containerID="0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c"
Jul 28 18:00:05.231353 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.231333    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"]
Jul 28 18:00:05.231760 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.231747    2579 scope.go:117] "RemoveContainer" containerID="73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d"
Jul 28 18:00:05.232039 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:00:05.232015    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d\": container with ID starting with 73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d not found: ID does not exist" containerID="73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d"
Jul 28 18:00:05.232123 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.232047    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d"} err="failed to get container status \"73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d\": rpc error: code = NotFound desc = could not find container \"73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d\": container with ID starting with 73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d not found: ID does not exist"
Jul 28 18:00:05.232123 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.232064    2579 scope.go:117] "RemoveContainer" containerID="0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c"
Jul 28 18:00:05.232327 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:00:05.232311    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c\": container with ID starting with 0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c not found: ID does not exist" containerID="0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c"
Jul 28 18:00:05.232367 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.232332    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c"} err="failed to get container status \"0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c\": rpc error: code = NotFound desc = could not find container \"0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c\": container with ID starting with 0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c not found: ID does not exist"
Jul 28 18:00:05.232367 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.232346    2579 scope.go:117] "RemoveContainer" containerID="73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d"
Jul 28 18:00:05.232565 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.232549    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d"} err="failed to get container status \"73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d\": rpc error: code = NotFound desc = could not find container \"73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d\": container with ID starting with 73b523fff0560971c9928c3f9376517441fb9d8c3a5679c6f2cffe7cc139c55d not found: ID does not exist"
Jul 28 18:00:05.232603 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.232566    2579 scope.go:117] "RemoveContainer" containerID="0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c"
Jul 28 18:00:05.232762 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.232747    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c"} err="failed to get container status \"0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c\": rpc error: code = NotFound desc = could not find container \"0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c\": container with ID starting with 0376483cdf4f970d3909709ed05bc875fdcbe27232abe69919aa76c76f673a1c not found: ID does not exist"
Jul 28 18:00:05.235543 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.235523    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-6d9fffb56c944fl"]
Jul 28 18:00:05.246067 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.246048    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"]
Jul 28 18:00:05.250063 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:05.250040    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7b2jwxv"]
Jul 28 18:00:06.319616 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:06.319579    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" path="/var/lib/kubelet/pods/8b33e9ac-31d4-4954-8a07-80e7f4e41bcb/volumes"
Jul 28 18:00:06.320193 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:06.320173    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" path="/var/lib/kubelet/pods/a4ec3e8c-6f13-41ce-96b5-677757a0d026/volumes"
Jul 28 18:00:11.994260 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.994225    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"]
Jul 28 18:00:11.994679 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.994606    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main"
Jul 28 18:00:11.994679 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.994616    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main"
Jul 28 18:00:11.994679 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.994647    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="llm-d-routing-sidecar"
Jul 28 18:00:11.994679 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.994654    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="llm-d-routing-sidecar"
Jul 28 18:00:11.994679 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.994661    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main"
Jul 28 18:00:11.994679 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.994667    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main"
Jul 28 18:00:11.994908 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.994726    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="main"
Jul 28 18:00:11.994908 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.994737    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b33e9ac-31d4-4954-8a07-80e7f4e41bcb" containerName="main"
Jul 28 18:00:11.994908 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.994744    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ec3e8c-6f13-41ce-96b5-677757a0d026" containerName="llm-d-routing-sidecar"
Jul 28 18:00:11.998734 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:11.998715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:12.001445 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.001426    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 18:00:12.002519 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.002503    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 18:00:12.002565 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.002506    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-9d7kz\""
Jul 28 18:00:12.005115 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.005091    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"]
Jul 28 18:00:12.062325 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.062292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gp85\" (UniqueName: \"kubernetes.io/projected/bb595ccd-e024-40db-b257-c1ec0b94bf65-kube-api-access-7gp85\") pod \"e2e-pvc-model-download-4q52x\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:12.163458 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.163423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gp85\" (UniqueName: \"kubernetes.io/projected/bb595ccd-e024-40db-b257-c1ec0b94bf65-kube-api-access-7gp85\") pod \"e2e-pvc-model-download-4q52x\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:12.172645 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.172619    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 18:00:12.183520 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.183500    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 18:00:12.193966 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.193939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gp85\" (UniqueName: \"kubernetes.io/projected/bb595ccd-e024-40db-b257-c1ec0b94bf65-kube-api-access-7gp85\") pod \"e2e-pvc-model-download-4q52x\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:12.459889 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.459862    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:00:12.469861 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.469838    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:00:12.685563 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:12.685534    2579 scope.go:117] "RemoveContainer" containerID="a4f152ffcdd0dc58c9842cf0b53dd9cc52c13545d81d24f41327ca4b57712641"
Jul 28 18:00:13.723230 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:13.723196    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"]
Jul 28 18:00:13.727037 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:13.727012    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-48639af5/e2e-pvc-model-download-rv8z8"]
Jul 28 18:00:14.320128 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:14.320073    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0265ec5-01a4-40e9-9cab-64832ae67a91" path="/var/lib/kubelet/pods/e0265ec5-01a4-40e9-9cab-64832ae67a91/volumes"
Jul 28 18:00:21.440232 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:21.440189    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"e2e-pvc-model-download-4q52x\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:21.446471 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:21.446445    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"e2e-pvc-model-download-4q52x\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:21.541694 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:21.541663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"e2e-pvc-model-download-4q52x\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:21.541841 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:21.541802    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"e2e-pvc-model-download-4q52x\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:21.545731 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:21.545710    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"e2e-pvc-model-download-4q52x\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") DevicePath \"csi-9ab66d4e0a31a3f0872af412be8bff1aa8ee85a50224e6baa8140dd1e93a8c41\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:21.778346 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:21.778315    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"e2e-pvc-model-download-4q52x\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/bb144e5b3a364e155f00fbf919f9353d1106d5e299f9e2d078b7d848700d710e/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:21.793657 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:21.793628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"e2e-pvc-model-download-4q52x\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") " pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:21.912198 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:21.912170    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-9d7kz\""
Jul 28 18:00:21.920106 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:21.920086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:22.053146 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:22.053120    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"]
Jul 28 18:00:22.054878 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:00:22.054847    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb595ccd_e024_40db_b257_c1ec0b94bf65.slice/crio-24044662f1edfc3709786cbcee4db604a7d13ffc1adad99661873efb753dd501 WatchSource:0}: Error finding container 24044662f1edfc3709786cbcee4db604a7d13ffc1adad99661873efb753dd501: Status 404 returned error can't find the container with id 24044662f1edfc3709786cbcee4db604a7d13ffc1adad99661873efb753dd501
Jul 28 18:00:22.294550 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:22.294516    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x" event={"ID":"bb595ccd-e024-40db-b257-c1ec0b94bf65","Type":"ContainerStarted","Data":"575b45218d53d8d2cb64cfa5f337ecca23d28127d47d49a3d35fd1cb2a28562b"}
Jul 28 18:00:22.294700 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:22.294557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x" event={"ID":"bb595ccd-e024-40db-b257-c1ec0b94bf65","Type":"ContainerStarted","Data":"24044662f1edfc3709786cbcee4db604a7d13ffc1adad99661873efb753dd501"}
Jul 28 18:00:22.317520 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:22.317470    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x" podStartSLOduration=14.317454767 podStartE2EDuration="14.317454767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:22.315047406 +0000 UTC m=+3010.544246526" watchObservedRunningTime="2026-07-28 18:00:22.317454767 +0000 UTC m=+3010.546653862"
Jul 28 18:00:32.346476 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:32.346443    2579 generic.go:356] "Generic (PLEG): container finished" podID="bb595ccd-e024-40db-b257-c1ec0b94bf65" containerID="575b45218d53d8d2cb64cfa5f337ecca23d28127d47d49a3d35fd1cb2a28562b" exitCode=0
Jul 28 18:00:32.346963 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:32.346518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x" event={"ID":"bb595ccd-e024-40db-b257-c1ec0b94bf65","Type":"ContainerDied","Data":"575b45218d53d8d2cb64cfa5f337ecca23d28127d47d49a3d35fd1cb2a28562b"}
Jul 28 18:00:33.508474 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:33.508448    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:33.646409 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:33.646334    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gp85\" (UniqueName: \"kubernetes.io/projected/bb595ccd-e024-40db-b257-c1ec0b94bf65-kube-api-access-7gp85\") pod \"bb595ccd-e024-40db-b257-c1ec0b94bf65\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") "
Jul 28 18:00:33.646563 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:33.646479    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"bb595ccd-e024-40db-b257-c1ec0b94bf65\" (UID: \"bb595ccd-e024-40db-b257-c1ec0b94bf65\") "
Jul 28 18:00:33.648649 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:33.648619    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb595ccd-e024-40db-b257-c1ec0b94bf65-kube-api-access-7gp85" (OuterVolumeSpecName: "kube-api-access-7gp85") pod "bb595ccd-e024-40db-b257-c1ec0b94bf65" (UID: "bb595ccd-e024-40db-b257-c1ec0b94bf65"). InnerVolumeSpecName "kube-api-access-7gp85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:00:33.649338 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:33.649317    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a" (OuterVolumeSpecName: "model-storage") pod "bb595ccd-e024-40db-b257-c1ec0b94bf65" (UID: "bb595ccd-e024-40db-b257-c1ec0b94bf65"). InnerVolumeSpecName "pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:00:33.747808 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:33.747769    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gp85\" (UniqueName: \"kubernetes.io/projected/bb595ccd-e024-40db-b257-c1ec0b94bf65-kube-api-access-7gp85\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:00:33.747971 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:33.747819    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") on node \"ip-10-0-136-111.ec2.internal\" "
Jul 28 18:00:33.875157 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:33.875134    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a") on node "ip-10-0-136-111.ec2.internal"
Jul 28 18:00:33.948799 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:33.948725    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"csi-9ab66d4e0a31a3f0872af412be8bff1aa8ee85a50224e6baa8140dd1e93a8c41\""
Jul 28 18:00:34.358038 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:34.358011    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"
Jul 28 18:00:34.358038 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:34.358023    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x" event={"ID":"bb595ccd-e024-40db-b257-c1ec0b94bf65","Type":"ContainerDied","Data":"24044662f1edfc3709786cbcee4db604a7d13ffc1adad99661873efb753dd501"}
Jul 28 18:00:34.358261 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:34.358054    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24044662f1edfc3709786cbcee4db604a7d13ffc1adad99661873efb753dd501"
Jul 28 18:00:49.837018 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.836980    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:00:49.837462 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.837367    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb595ccd-e024-40db-b257-c1ec0b94bf65" containerName="storage-initializer"
Jul 28 18:00:49.837462 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.837379    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb595ccd-e024-40db-b257-c1ec0b94bf65" containerName="storage-initializer"
Jul 28 18:00:49.837462 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.837443    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb595ccd-e024-40db-b257-c1ec0b94bf65" containerName="storage-initializer"
Jul 28 18:00:49.844603 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.844575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:49.851899 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.851875    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 28 18:00:49.869423 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.869394    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-6lzqk\""
Jul 28 18:00:49.869423 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.869413    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 28 18:00:49.869595 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.869507    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 28 18:00:49.869595 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.869520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-9d7kz\""
Jul 28 18:00:49.871631 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.871608    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:00:49.995731 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.995698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:49.995901 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.995750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:49.995901 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.995817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:49.995901 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.995875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc6632b-31c9-4a07-9231-97758249ee8c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:49.996005 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.995939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47pdq\" (UniqueName: \"kubernetes.io/projected/0fc6632b-31c9-4a07-9231-97758249ee8c-kube-api-access-47pdq\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:49.996005 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:49.995973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.097562 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.097464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.097736 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.097602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc6632b-31c9-4a07-9231-97758249ee8c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.097736 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.097676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47pdq\" (UniqueName: \"kubernetes.io/projected/0fc6632b-31c9-4a07-9231-97758249ee8c-kube-api-access-47pdq\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.097736 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.097710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.097915 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.097750    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.097915 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.097819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.098158 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.098139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.098256 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.098176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.098319 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.098274    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.099877 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.099857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.099981 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.099963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc6632b-31c9-4a07-9231-97758249ee8c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.111640 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.111617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47pdq\" (UniqueName: \"kubernetes.io/projected/0fc6632b-31c9-4a07-9231-97758249ee8c-kube-api-access-47pdq\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:50.114242 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.114216    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"]
Jul 28 18:00:50.117697 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.117683    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:00:50.120593 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.120571    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-vtc68\""
Jul 28 18:00:50.129158 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.129135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"]
Jul 28 18:00:50.199153 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.199123    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h6rg\" (UniqueName: \"kubernetes.io/projected/203eadd7-2be1-402b-bb24-f4257ffabe49-kube-api-access-4h6rg\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m\" (UID: \"203eadd7-2be1-402b-bb24-f4257ffabe49\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:00:50.199319 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.199174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/203eadd7-2be1-402b-bb24-f4257ffabe49-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m\" (UID: \"203eadd7-2be1-402b-bb24-f4257ffabe49\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:00:50.300320 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.300296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h6rg\" (UniqueName: \"kubernetes.io/projected/203eadd7-2be1-402b-bb24-f4257ffabe49-kube-api-access-4h6rg\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m\" (UID: \"203eadd7-2be1-402b-bb24-f4257ffabe49\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:00:50.300413 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.300340    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/203eadd7-2be1-402b-bb24-f4257ffabe49-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m\" (UID: \"203eadd7-2be1-402b-bb24-f4257ffabe49\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:00:50.302740 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.302714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/203eadd7-2be1-402b-bb24-f4257ffabe49-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m\" (UID: \"203eadd7-2be1-402b-bb24-f4257ffabe49\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:00:50.311390 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.311363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h6rg\" (UniqueName: \"kubernetes.io/projected/203eadd7-2be1-402b-bb24-f4257ffabe49-kube-api-access-4h6rg\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m\" (UID: \"203eadd7-2be1-402b-bb24-f4257ffabe49\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:00:50.431434 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.431348    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:00:50.569011 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:50.568987    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"]
Jul 28 18:00:50.571060 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:00:50.571032    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203eadd7_2be1_402b_bb24_f4257ffabe49.slice/crio-15ad98ea17bdea613463fc5b76f99646c88e2b373a84a8ba2f9ce10dca783590 WatchSource:0}: Error finding container 15ad98ea17bdea613463fc5b76f99646c88e2b373a84a8ba2f9ce10dca783590: Status 404 returned error can't find the container with id 15ad98ea17bdea613463fc5b76f99646c88e2b373a84a8ba2f9ce10dca783590
Jul 28 18:00:51.437874 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:51.437841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m" event={"ID":"203eadd7-2be1-402b-bb24-f4257ffabe49","Type":"ContainerStarted","Data":"3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b"}
Jul 28 18:00:51.438299 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:51.437891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m" event={"ID":"203eadd7-2be1-402b-bb24-f4257ffabe49","Type":"ContainerStarted","Data":"15ad98ea17bdea613463fc5b76f99646c88e2b373a84a8ba2f9ce10dca783590"}
Jul 28 18:00:51.438299 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:51.437910    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:00:51.458870 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:51.458827    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m" podStartSLOduration=1.4588115369999999 podStartE2EDuration="1.458811537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:51.456571402 +0000 UTC m=+3039.685770497" watchObservedRunningTime="2026-07-28 18:00:51.458811537 +0000 UTC m=+3039.688010632"
Jul 28 18:00:52.116766 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:52.116735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:52.124549 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:52.124514    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") device path: \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:52.218098 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:52.218052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:52.218225 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:52.218181    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") DevicePath \"\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:52.221522 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:52.221503    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") DevicePath \"csi-9ab66d4e0a31a3f0872af412be8bff1aa8ee85a50224e6baa8140dd1e93a8c41\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:52.307850 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:52.307819    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/bb144e5b3a364e155f00fbf919f9353d1106d5e299f9e2d078b7d848700d710e/globalmount\"" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:52.320673 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:52.320645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:52.554738 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:52.554645    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:00:52.696270 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:52.696242    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:00:52.698383 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:00:52.698354    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc6632b_31c9_4a07_9231_97758249ee8c.slice/crio-3506ab3f4b5581be1bf373a42e88cbbd6e63b8f0eb0cb274972d2a6871206d90 WatchSource:0}: Error finding container 3506ab3f4b5581be1bf373a42e88cbbd6e63b8f0eb0cb274972d2a6871206d90: Status 404 returned error can't find the container with id 3506ab3f4b5581be1bf373a42e88cbbd6e63b8f0eb0cb274972d2a6871206d90
Jul 28 18:00:53.451318 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:53.451276    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"0fc6632b-31c9-4a07-9231-97758249ee8c","Type":"ContainerStarted","Data":"a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929"}
Jul 28 18:00:53.451548 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:53.451345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"0fc6632b-31c9-4a07-9231-97758249ee8c","Type":"ContainerStarted","Data":"3506ab3f4b5581be1bf373a42e88cbbd6e63b8f0eb0cb274972d2a6871206d90"}
Jul 28 18:00:53.473459 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:53.473399    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=4.473382759 podStartE2EDuration="4.473382759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:53.472693401 +0000 UTC m=+3041.701892498" watchObservedRunningTime="2026-07-28 18:00:53.473382759 +0000 UTC m=+3041.702581858"
Jul 28 18:00:53.555838 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:00:53.555801    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:01:02.555246 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:01:02.555213    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:01:02.556708 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:01:02.556678    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 28 18:01:12.555483 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:01:12.555391    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 28 18:01:22.444763 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:01:22.444649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:01:22.555346 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:01:22.555305    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 28 18:01:32.556071 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:01:32.556025    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 28 18:01:42.555463 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:01:42.555421    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 28 18:01:52.555709 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:01:52.555669    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 28 18:02:02.555198 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:02.555143    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 28 18:02:12.555807 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:12.555762    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 28 18:02:22.555818 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:22.555776    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 28 18:02:32.565356 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:32.565324    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:02:32.573019 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:32.572994    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:02:44.291287 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:44.291206    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"]
Jul 28 18:02:44.297171 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:44.297140    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:02:52.444395 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:52.444356    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m" podUID="203eadd7-2be1-402b-bb24-f4257ffabe49" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:02:59.310870 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:59.310823    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m" podUID="203eadd7-2be1-402b-bb24-f4257ffabe49" containerName="main" containerID="cri-o://3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b" gracePeriod=45
Jul 28 18:02:59.320179 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:59.319984    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" containerID="cri-o://a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929" gracePeriod=45
Jul 28 18:02:59.565970 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:59.565938    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.73:8000/health\": dial tcp 10.132.0.73:8000: connect: connection refused"
Jul 28 18:02:59.567219 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:59.567200    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:02:59.621225 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:59.621189    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/203eadd7-2be1-402b-bb24-f4257ffabe49-tls-certs\") pod \"203eadd7-2be1-402b-bb24-f4257ffabe49\" (UID: \"203eadd7-2be1-402b-bb24-f4257ffabe49\") "
Jul 28 18:02:59.621430 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:59.621264    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4h6rg\" (UniqueName: \"kubernetes.io/projected/203eadd7-2be1-402b-bb24-f4257ffabe49-kube-api-access-4h6rg\") pod \"203eadd7-2be1-402b-bb24-f4257ffabe49\" (UID: \"203eadd7-2be1-402b-bb24-f4257ffabe49\") "
Jul 28 18:02:59.623310 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:59.623280    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/203eadd7-2be1-402b-bb24-f4257ffabe49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "203eadd7-2be1-402b-bb24-f4257ffabe49" (UID: "203eadd7-2be1-402b-bb24-f4257ffabe49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:02:59.623471 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:59.623456    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203eadd7-2be1-402b-bb24-f4257ffabe49-kube-api-access-4h6rg" (OuterVolumeSpecName: "kube-api-access-4h6rg") pod "203eadd7-2be1-402b-bb24-f4257ffabe49" (UID: "203eadd7-2be1-402b-bb24-f4257ffabe49"). InnerVolumeSpecName "kube-api-access-4h6rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:02:59.721875 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:59.721842    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/203eadd7-2be1-402b-bb24-f4257ffabe49-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:02:59.721875 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:02:59.721873    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h6rg\" (UniqueName: \"kubernetes.io/projected/203eadd7-2be1-402b-bb24-f4257ffabe49-kube-api-access-4h6rg\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:00.065472 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.065446    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:03:00.094971 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.094891    2579 generic.go:356] "Generic (PLEG): container finished" podID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerID="a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929" exitCode=0
Jul 28 18:03:00.094971 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.094935    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 28 18:03:00.094971 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.094946    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"0fc6632b-31c9-4a07-9231-97758249ee8c","Type":"ContainerDied","Data":"a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929"}
Jul 28 18:03:00.095236 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.095000    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"0fc6632b-31c9-4a07-9231-97758249ee8c","Type":"ContainerDied","Data":"3506ab3f4b5581be1bf373a42e88cbbd6e63b8f0eb0cb274972d2a6871206d90"}
Jul 28 18:03:00.095236 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.095024    2579 scope.go:117] "RemoveContainer" containerID="a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929"
Jul 28 18:03:00.096395 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.096375    2579 generic.go:356] "Generic (PLEG): container finished" podID="203eadd7-2be1-402b-bb24-f4257ffabe49" containerID="3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b" exitCode=0
Jul 28 18:03:00.096514 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.096405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m" event={"ID":"203eadd7-2be1-402b-bb24-f4257ffabe49","Type":"ContainerDied","Data":"3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b"}
Jul 28 18:03:00.096514 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.096467    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m" event={"ID":"203eadd7-2be1-402b-bb24-f4257ffabe49","Type":"ContainerDied","Data":"15ad98ea17bdea613463fc5b76f99646c88e2b373a84a8ba2f9ce10dca783590"}
Jul 28 18:03:00.096514 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.096415    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"
Jul 28 18:03:00.103783 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.103767    2579 scope.go:117] "RemoveContainer" containerID="a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929"
Jul 28 18:03:00.104024 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:03:00.104005    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929\": container with ID starting with a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929 not found: ID does not exist" containerID="a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929"
Jul 28 18:03:00.104115 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.104031    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929"} err="failed to get container status \"a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929\": rpc error: code = NotFound desc = could not find container \"a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929\": container with ID starting with a0416d1231af7b624a376cbb8257deb75e8653515fa025ed4a2587ff856ff929 not found: ID does not exist"
Jul 28 18:03:00.104115 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.104046    2579 scope.go:117] "RemoveContainer" containerID="3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b"
Jul 28 18:03:00.112060 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.112044    2579 scope.go:117] "RemoveContainer" containerID="3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b"
Jul 28 18:03:00.112364 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:03:00.112342    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b\": container with ID starting with 3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b not found: ID does not exist" containerID="3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b"
Jul 28 18:03:00.112440 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.112373    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b"} err="failed to get container status \"3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b\": rpc error: code = NotFound desc = could not find container \"3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b\": container with ID starting with 3d0b635e4e65bb8f844378c7c162300c6c63947dc3e3286b05f812207faf8e5b not found: ID does not exist"
Jul 28 18:03:00.123177 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.123154    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"]
Jul 28 18:03:00.125681 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.125665    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc6632b-31c9-4a07-9231-97758249ee8c-tls-certs\") pod \"0fc6632b-31c9-4a07-9231-97758249ee8c\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") "
Jul 28 18:03:00.125787 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.125771    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") pod \"0fc6632b-31c9-4a07-9231-97758249ee8c\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") "
Jul 28 18:03:00.125855 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.125815    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-model-cache\") pod \"0fc6632b-31c9-4a07-9231-97758249ee8c\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") "
Jul 28 18:03:00.125855 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.125849    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-home\") pod \"0fc6632b-31c9-4a07-9231-97758249ee8c\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") "
Jul 28 18:03:00.125966 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.125891    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-tmp-dir\") pod \"0fc6632b-31c9-4a07-9231-97758249ee8c\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") "
Jul 28 18:03:00.125966 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.125954    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-dshm\") pod \"0fc6632b-31c9-4a07-9231-97758249ee8c\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") "
Jul 28 18:03:00.126069 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.125987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47pdq\" (UniqueName: \"kubernetes.io/projected/0fc6632b-31c9-4a07-9231-97758249ee8c-kube-api-access-47pdq\") pod \"0fc6632b-31c9-4a07-9231-97758249ee8c\" (UID: \"0fc6632b-31c9-4a07-9231-97758249ee8c\") "
Jul 28 18:03:00.126356 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.126097    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0fc6632b-31c9-4a07-9231-97758249ee8c" (UID: "0fc6632b-31c9-4a07-9231-97758249ee8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:00.126356 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.126286    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:00.126549 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.126525    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-home" (OuterVolumeSpecName: "home") pod "0fc6632b-31c9-4a07-9231-97758249ee8c" (UID: "0fc6632b-31c9-4a07-9231-97758249ee8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:00.128369 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.128316    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc6632b-31c9-4a07-9231-97758249ee8c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0fc6632b-31c9-4a07-9231-97758249ee8c" (UID: "0fc6632b-31c9-4a07-9231-97758249ee8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:03:00.128685 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.128659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc6632b-31c9-4a07-9231-97758249ee8c-kube-api-access-47pdq" (OuterVolumeSpecName: "kube-api-access-47pdq") pod "0fc6632b-31c9-4a07-9231-97758249ee8c" (UID: "0fc6632b-31c9-4a07-9231-97758249ee8c"). InnerVolumeSpecName "kube-api-access-47pdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:03:00.128822 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.128803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a" (OuterVolumeSpecName: "kserve-pvc-source") pod "0fc6632b-31c9-4a07-9231-97758249ee8c" (UID: "0fc6632b-31c9-4a07-9231-97758249ee8c"). InnerVolumeSpecName "pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 28 18:03:00.129370 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.129346    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-dshm" (OuterVolumeSpecName: "dshm") pod "0fc6632b-31c9-4a07-9231-97758249ee8c" (UID: "0fc6632b-31c9-4a07-9231-97758249ee8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:00.129908 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.129891    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schemcj5m"]
Jul 28 18:03:00.137481 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.137457    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0fc6632b-31c9-4a07-9231-97758249ee8c" (UID: "0fc6632b-31c9-4a07-9231-97758249ee8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:00.227496 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.227450    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47pdq\" (UniqueName: \"kubernetes.io/projected/0fc6632b-31c9-4a07-9231-97758249ee8c-kube-api-access-47pdq\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:00.227496 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.227490    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fc6632b-31c9-4a07-9231-97758249ee8c-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:00.227715 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.227522    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") on node \"ip-10-0-136-111.ec2.internal\" "
Jul 28 18:03:00.227715 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.227532    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:00.227715 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.227542    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:00.227715 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.227550    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fc6632b-31c9-4a07-9231-97758249ee8c-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:00.256061 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.256035    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a") on node "ip-10-0-136-111.ec2.internal"
Jul 28 18:03:00.320470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.320439    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="203eadd7-2be1-402b-bb24-f4257ffabe49" path="/var/lib/kubelet/pods/203eadd7-2be1-402b-bb24-f4257ffabe49/volumes"
Jul 28 18:03:00.328100 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.328053    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-a4971758-a37e-4b58-bdc5-15dd9b40c734\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0690616aaf6246a0a\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"csi-9ab66d4e0a31a3f0872af412be8bff1aa8ee85a50224e6baa8140dd1e93a8c41\""
Jul 28 18:03:00.412129 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.412032    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:03:00.415958 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:00.415932    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 28 18:03:02.320373 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:02.320340    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" path="/var/lib/kubelet/pods/0fc6632b-31c9-4a07-9231-97758249ee8c/volumes"
Jul 28 18:03:10.766483 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:10.766448    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"]
Jul 28 18:03:10.770164 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:10.770138    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/e2e-pvc-model-download-4q52x"]
Jul 28 18:03:12.321468 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:12.321435    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb595ccd-e024-40db-b257-c1ec0b94bf65" path="/var/lib/kubelet/pods/bb595ccd-e024-40db-b257-c1ec0b94bf65/volumes"
Jul 28 18:03:12.741113 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:12.741003    2579 scope.go:117] "RemoveContainer" containerID="046a7dee40a106a8e0962838f282e47ed582b73599b2005b74f23215ace833b5"
Jul 28 18:03:21.786283 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.786250    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"]
Jul 28 18:03:21.786825 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.786805    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203eadd7-2be1-402b-bb24-f4257ffabe49" containerName="main"
Jul 28 18:03:21.786907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.786828    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="203eadd7-2be1-402b-bb24-f4257ffabe49" containerName="main"
Jul 28 18:03:21.786907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.786870    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main"
Jul 28 18:03:21.786907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.786880    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main"
Jul 28 18:03:21.787074 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.786971    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc6632b-31c9-4a07-9231-97758249ee8c" containerName="main"
Jul 28 18:03:21.787074 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.786990    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="203eadd7-2be1-402b-bb24-f4257ffabe49" containerName="main"
Jul 28 18:03:21.791093 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.791058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:21.794199 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.794176    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-kserve-self-signed-certs\""
Jul 28 18:03:21.794330 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.794175    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"openshift-service-ca.crt\""
Jul 28 18:03:21.794330 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.794269    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"seaweedfs-s3-creds\""
Jul 28 18:03:21.794680 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.794660    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"kube-root-ca.crt\""
Jul 28 18:03:21.795054 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.795036    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"default-dockercfg-6md7b\""
Jul 28 18:03:21.808359 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.808337    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"]
Jul 28 18:03:21.918526 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.918490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-model-cache\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:21.918795 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.918551    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:21.918795 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.918571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdm7g\" (UniqueName: \"kubernetes.io/projected/dea10d68-a9b3-433a-aa5f-953d65d490dd-kube-api-access-bdm7g\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:21.918795 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.918606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-home\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:21.918795 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.918631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-dshm\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:21.918795 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.918654    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:21.918795 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:21.918696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea10d68-a9b3-433a-aa5f-953d65d490dd-tls-certs\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.020241 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.020197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.020424 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.020322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdm7g\" (UniqueName: \"kubernetes.io/projected/dea10d68-a9b3-433a-aa5f-953d65d490dd-kube-api-access-bdm7g\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.020424 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.020395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-home\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.020550 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.020435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-dshm\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.020550 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.020460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.020550 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.020499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea10d68-a9b3-433a-aa5f-953d65d490dd-tls-certs\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.020550 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.020528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-model-cache\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.020965 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.020939    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-model-cache\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.021054 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.021031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-tmp-dir\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.021151 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.021097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-home\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.021244 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.021223    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-kserve-provision-location\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.023283 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.023260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-dshm\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.023548 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.023530    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea10d68-a9b3-433a-aa5f-953d65d490dd-tls-certs\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.031148 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.031124    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"]
Jul 28 18:03:22.034360 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.034340    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:22.035984 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.035961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdm7g\" (UniqueName: \"kubernetes.io/projected/dea10d68-a9b3-433a-aa5f-953d65d490dd-kube-api-access-bdm7g\") pod \"e2e-del-blocked-svc-kserve-687558855-vhdrk\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.037379 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.037325    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-config-deletion-blocked-when-referenced-0f029b85\"/\"e2e-del-blocked-svc-epp-sa-dockercfg-hbw28\""
Jul 28 18:03:22.050271 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.050247    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"]
Jul 28 18:03:22.101523 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.101485    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:22.121317 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.121283    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5\" (UID: \"01f44f4b-1f2b-4e6e-af48-6aa287b10d00\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:22.121465 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.121410    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwhvq\" (UniqueName: \"kubernetes.io/projected/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-kube-api-access-nwhvq\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5\" (UID: \"01f44f4b-1f2b-4e6e-af48-6aa287b10d00\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:22.222872 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.222838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwhvq\" (UniqueName: \"kubernetes.io/projected/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-kube-api-access-nwhvq\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5\" (UID: \"01f44f4b-1f2b-4e6e-af48-6aa287b10d00\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:22.223032 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.222898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5\" (UID: \"01f44f4b-1f2b-4e6e-af48-6aa287b10d00\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:22.225443 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.225420    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-tls-certs\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5\" (UID: \"01f44f4b-1f2b-4e6e-af48-6aa287b10d00\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:22.231119 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.231073    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwhvq\" (UniqueName: \"kubernetes.io/projected/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-kube-api-access-nwhvq\") pod \"e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5\" (UID: \"01f44f4b-1f2b-4e6e-af48-6aa287b10d00\") " pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:22.238273 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.238246    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"]
Jul 28 18:03:22.241768 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:03:22.241740    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea10d68_a9b3_433a_aa5f_953d65d490dd.slice/crio-22c47865d91fe4332207f39747015a10530183f6e267f0ffd64f891c7b2599da WatchSource:0}: Error finding container 22c47865d91fe4332207f39747015a10530183f6e267f0ffd64f891c7b2599da: Status 404 returned error can't find the container with id 22c47865d91fe4332207f39747015a10530183f6e267f0ffd64f891c7b2599da
Jul 28 18:03:22.356550 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.356519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:22.498777 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:22.498747    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"]
Jul 28 18:03:22.501618 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:03:22.501590    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f44f4b_1f2b_4e6e_af48_6aa287b10d00.slice/crio-62ec40128884b3e3f6be92aa78d953308bbdb9dfef8f7fc47481b4d591e0b292 WatchSource:0}: Error finding container 62ec40128884b3e3f6be92aa78d953308bbdb9dfef8f7fc47481b4d591e0b292: Status 404 returned error can't find the container with id 62ec40128884b3e3f6be92aa78d953308bbdb9dfef8f7fc47481b4d591e0b292
Jul 28 18:03:23.219590 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:23.219550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5" event={"ID":"01f44f4b-1f2b-4e6e-af48-6aa287b10d00","Type":"ContainerStarted","Data":"8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac"}
Jul 28 18:03:23.220056 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:23.219621    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5" event={"ID":"01f44f4b-1f2b-4e6e-af48-6aa287b10d00","Type":"ContainerStarted","Data":"62ec40128884b3e3f6be92aa78d953308bbdb9dfef8f7fc47481b4d591e0b292"}
Jul 28 18:03:23.220056 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:23.219656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:23.221060 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:23.221036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk" event={"ID":"dea10d68-a9b3-433a-aa5f-953d65d490dd","Type":"ContainerStarted","Data":"415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b"}
Jul 28 18:03:23.221214 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:23.221086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk" event={"ID":"dea10d68-a9b3-433a-aa5f-953d65d490dd","Type":"ContainerStarted","Data":"22c47865d91fe4332207f39747015a10530183f6e267f0ffd64f891c7b2599da"}
Jul 28 18:03:23.238536 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:23.238487    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5" podStartSLOduration=1.238472973 podStartE2EDuration="1.238472973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:23.237036384 +0000 UTC m=+3191.466235480" watchObservedRunningTime="2026-07-28 18:03:23.238472973 +0000 UTC m=+3191.467672090"
Jul 28 18:03:24.694666 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:24.694623    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"]
Jul 28 18:03:24.696624 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:24.696587    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"]
Jul 28 18:03:25.230745 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:25.230674    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk" podUID="dea10d68-a9b3-433a-aa5f-953d65d490dd" containerName="storage-initializer" containerID="cri-o://415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b" gracePeriod=60
Jul 28 18:03:27.398894 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.398860    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:27.470042 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.469954    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdm7g\" (UniqueName: \"kubernetes.io/projected/dea10d68-a9b3-433a-aa5f-953d65d490dd-kube-api-access-bdm7g\") pod \"dea10d68-a9b3-433a-aa5f-953d65d490dd\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") "
Jul 28 18:03:27.470221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470092    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-dshm\") pod \"dea10d68-a9b3-433a-aa5f-953d65d490dd\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") "
Jul 28 18:03:27.470221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-model-cache\") pod \"dea10d68-a9b3-433a-aa5f-953d65d490dd\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") "
Jul 28 18:03:27.470221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470134    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-home\") pod \"dea10d68-a9b3-433a-aa5f-953d65d490dd\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") "
Jul 28 18:03:27.470221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470150    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-kserve-provision-location\") pod \"dea10d68-a9b3-433a-aa5f-953d65d490dd\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") "
Jul 28 18:03:27.470221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470172    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea10d68-a9b3-433a-aa5f-953d65d490dd-tls-certs\") pod \"dea10d68-a9b3-433a-aa5f-953d65d490dd\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") "
Jul 28 18:03:27.470221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470208    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-tmp-dir\") pod \"dea10d68-a9b3-433a-aa5f-953d65d490dd\" (UID: \"dea10d68-a9b3-433a-aa5f-953d65d490dd\") "
Jul 28 18:03:27.470515 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470325    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "dea10d68-a9b3-433a-aa5f-953d65d490dd" (UID: "dea10d68-a9b3-433a-aa5f-953d65d490dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:27.470515 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470380    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-home" (OuterVolumeSpecName: "home") pod "dea10d68-a9b3-433a-aa5f-953d65d490dd" (UID: "dea10d68-a9b3-433a-aa5f-953d65d490dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:27.470613 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470514    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dea10d68-a9b3-433a-aa5f-953d65d490dd" (UID: "dea10d68-a9b3-433a-aa5f-953d65d490dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:27.470613 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470531    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:27.470613 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.470548    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:27.472854 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.472804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-dshm" (OuterVolumeSpecName: "dshm") pod "dea10d68-a9b3-433a-aa5f-953d65d490dd" (UID: "dea10d68-a9b3-433a-aa5f-953d65d490dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:27.472982 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.472896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea10d68-a9b3-433a-aa5f-953d65d490dd-kube-api-access-bdm7g" (OuterVolumeSpecName: "kube-api-access-bdm7g") pod "dea10d68-a9b3-433a-aa5f-953d65d490dd" (UID: "dea10d68-a9b3-433a-aa5f-953d65d490dd"). InnerVolumeSpecName "kube-api-access-bdm7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:03:27.472982 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.472922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea10d68-a9b3-433a-aa5f-953d65d490dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dea10d68-a9b3-433a-aa5f-953d65d490dd" (UID: "dea10d68-a9b3-433a-aa5f-953d65d490dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:03:27.548808 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.548760    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dea10d68-a9b3-433a-aa5f-953d65d490dd" (UID: "dea10d68-a9b3-433a-aa5f-953d65d490dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:03:27.571177 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.571137    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:27.571177 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.571169    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea10d68-a9b3-433a-aa5f-953d65d490dd-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:27.571177 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.571180    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:27.571460 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.571190    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdm7g\" (UniqueName: \"kubernetes.io/projected/dea10d68-a9b3-433a-aa5f-953d65d490dd-kube-api-access-bdm7g\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:27.571460 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:27.571198    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea10d68-a9b3-433a-aa5f-953d65d490dd-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:28.248675 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:28.248634    2579 generic.go:356] "Generic (PLEG): container finished" podID="dea10d68-a9b3-433a-aa5f-953d65d490dd" containerID="415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b" exitCode=0
Jul 28 18:03:28.248847 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:28.248682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk" event={"ID":"dea10d68-a9b3-433a-aa5f-953d65d490dd","Type":"ContainerDied","Data":"415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b"}
Jul 28 18:03:28.248847 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:28.248696    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"
Jul 28 18:03:28.248847 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:28.248756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk" event={"ID":"dea10d68-a9b3-433a-aa5f-953d65d490dd","Type":"ContainerDied","Data":"22c47865d91fe4332207f39747015a10530183f6e267f0ffd64f891c7b2599da"}
Jul 28 18:03:28.248847 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:28.248780    2579 scope.go:117] "RemoveContainer" containerID="415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b"
Jul 28 18:03:28.258384 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:28.258361    2579 scope.go:117] "RemoveContainer" containerID="415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b"
Jul 28 18:03:28.258625 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:03:28.258603    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b\": container with ID starting with 415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b not found: ID does not exist" containerID="415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b"
Jul 28 18:03:28.258701 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:28.258635    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b"} err="failed to get container status \"415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b\": rpc error: code = NotFound desc = could not find container \"415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b\": container with ID starting with 415114b6435cc649eee6e6db06e3ca43dc68a3b4c0c2f675771aa9db88778a1b not found: ID does not exist"
Jul 28 18:03:28.295788 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:28.295754    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"]
Jul 28 18:03:28.307525 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:28.307490    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-687558855-vhdrk"]
Jul 28 18:03:28.319580 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:28.319556    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dea10d68-a9b3-433a-aa5f-953d65d490dd" path="/var/lib/kubelet/pods/dea10d68-a9b3-433a-aa5f-953d65d490dd/volumes"
Jul 28 18:03:33.848273 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.848235    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"]
Jul 28 18:03:33.848682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.848629    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dea10d68-a9b3-433a-aa5f-953d65d490dd" containerName="storage-initializer"
Jul 28 18:03:33.848682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.848641    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dea10d68-a9b3-433a-aa5f-953d65d490dd" containerName="storage-initializer"
Jul 28 18:03:33.848755 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.848709    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dea10d68-a9b3-433a-aa5f-953d65d490dd" containerName="storage-initializer"
Jul 28 18:03:33.851840 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.851822    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:33.854744 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.854712    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 28 18:03:33.854912 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.854846    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 28 18:03:33.856181 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.856139    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-gvhzk\""
Jul 28 18:03:33.856364 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.856349    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 28 18:03:33.861618 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.861593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"]
Jul 28 18:03:33.930665 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.930634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rmg\" (UniqueName: \"kubernetes.io/projected/a886fdae-acc2-4dc1-b891-01b38bbf00e6-kube-api-access-26rmg\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:33.930812 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.930670    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:33.930812 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.930700    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:33.930812 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.930794    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:33.930912 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.930856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:33.930912 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:33.930887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.031494 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.031455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.031667 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.031502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.031667 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.031542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.031667 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.031571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.031803 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.031710    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rmg\" (UniqueName: \"kubernetes.io/projected/a886fdae-acc2-4dc1-b891-01b38bbf00e6-kube-api-access-26rmg\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.031803 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.031752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.032032 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.032008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.032032 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.032024    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.032214 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.032128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.033934 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.033911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.034040 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.034022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.040687 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.040664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rmg\" (UniqueName: \"kubernetes.io/projected/a886fdae-acc2-4dc1-b891-01b38bbf00e6-kube-api-access-26rmg\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.163145 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.163055    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:34.295240 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:34.295213    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"]
Jul 28 18:03:34.297302 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:03:34.297275    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda886fdae_acc2_4dc1_b891_01b38bbf00e6.slice/crio-7330fde2da18bf4d956f4b8baca96f91166c07f4a12cc09ff7be1af9840bf24c WatchSource:0}: Error finding container 7330fde2da18bf4d956f4b8baca96f91166c07f4a12cc09ff7be1af9840bf24c: Status 404 returned error can't find the container with id 7330fde2da18bf4d956f4b8baca96f91166c07f4a12cc09ff7be1af9840bf24c
Jul 28 18:03:35.291194 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:35.291157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp" event={"ID":"a886fdae-acc2-4dc1-b891-01b38bbf00e6","Type":"ContainerStarted","Data":"95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68"}
Jul 28 18:03:35.291596 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:35.291207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp" event={"ID":"a886fdae-acc2-4dc1-b891-01b38bbf00e6","Type":"ContainerStarted","Data":"7330fde2da18bf4d956f4b8baca96f91166c07f4a12cc09ff7be1af9840bf24c"}
Jul 28 18:03:35.316513 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:35.316463    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp" podStartSLOduration=2.316447488 podStartE2EDuration="2.316447488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:03:35.314488368 +0000 UTC m=+3203.543687514" watchObservedRunningTime="2026-07-28 18:03:35.316447488 +0000 UTC m=+3203.545646584"
Jul 28 18:03:36.163293 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:36.163260    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:40.249989 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:40.249931    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5" podUID="01f44f4b-1f2b-4e6e-af48-6aa287b10d00" containerName="main" containerID="cri-o://8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac" gracePeriod=45
Jul 28 18:03:40.507912 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:40.507846    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:40.590899 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:40.590868    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-tls-certs\") pod \"01f44f4b-1f2b-4e6e-af48-6aa287b10d00\" (UID: \"01f44f4b-1f2b-4e6e-af48-6aa287b10d00\") "
Jul 28 18:03:40.591071 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:40.590967    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwhvq\" (UniqueName: \"kubernetes.io/projected/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-kube-api-access-nwhvq\") pod \"01f44f4b-1f2b-4e6e-af48-6aa287b10d00\" (UID: \"01f44f4b-1f2b-4e6e-af48-6aa287b10d00\") "
Jul 28 18:03:40.593064 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:40.593035    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01f44f4b-1f2b-4e6e-af48-6aa287b10d00" (UID: "01f44f4b-1f2b-4e6e-af48-6aa287b10d00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:03:40.593182 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:40.593073    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-kube-api-access-nwhvq" (OuterVolumeSpecName: "kube-api-access-nwhvq") pod "01f44f4b-1f2b-4e6e-af48-6aa287b10d00" (UID: "01f44f4b-1f2b-4e6e-af48-6aa287b10d00"). InnerVolumeSpecName "kube-api-access-nwhvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:03:40.691819 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:40.691783    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:40.691819 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:40.691815    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwhvq\" (UniqueName: \"kubernetes.io/projected/01f44f4b-1f2b-4e6e-af48-6aa287b10d00-kube-api-access-nwhvq\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:03:41.320414 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:41.320381    2579 generic.go:356] "Generic (PLEG): container finished" podID="01f44f4b-1f2b-4e6e-af48-6aa287b10d00" containerID="8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac" exitCode=0
Jul 28 18:03:41.320876 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:41.320429    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"
Jul 28 18:03:41.320876 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:41.320437    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5" event={"ID":"01f44f4b-1f2b-4e6e-af48-6aa287b10d00","Type":"ContainerDied","Data":"8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac"}
Jul 28 18:03:41.320876 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:41.320484    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5" event={"ID":"01f44f4b-1f2b-4e6e-af48-6aa287b10d00","Type":"ContainerDied","Data":"62ec40128884b3e3f6be92aa78d953308bbdb9dfef8f7fc47481b4d591e0b292"}
Jul 28 18:03:41.320876 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:41.320508    2579 scope.go:117] "RemoveContainer" containerID="8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac"
Jul 28 18:03:41.329846 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:41.329825    2579 scope.go:117] "RemoveContainer" containerID="8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac"
Jul 28 18:03:41.330125 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:03:41.330104    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac\": container with ID starting with 8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac not found: ID does not exist" containerID="8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac"
Jul 28 18:03:41.330208 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:41.330138    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac"} err="failed to get container status \"8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac\": rpc error: code = NotFound desc = could not find container \"8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac\": container with ID starting with 8f98fed07776e1b5b473ee9e24126de76bee1399bb1eccd3bbce62912a4b16ac not found: ID does not exist"
Jul 28 18:03:41.343675 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:41.343636    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"]
Jul 28 18:03:41.349219 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:41.349197    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-config-deletion-blocked-when-referenced-0f029b85/e2e-del-blocked-svc-kserve-router-scheduler-5c74dd894f-rksc5"]
Jul 28 18:03:42.320410 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:42.320376    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01f44f4b-1f2b-4e6e-af48-6aa287b10d00" path="/var/lib/kubelet/pods/01f44f4b-1f2b-4e6e-af48-6aa287b10d00/volumes"
Jul 28 18:03:44.163593 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:44.163558    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:44.176199 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:44.176172    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:03:44.346797 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:03:44.346767    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:04:02.754624 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.754590    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"]
Jul 28 18:04:02.755091 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.754963    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01f44f4b-1f2b-4e6e-af48-6aa287b10d00" containerName="main"
Jul 28 18:04:02.755091 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.754974    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="01f44f4b-1f2b-4e6e-af48-6aa287b10d00" containerName="main"
Jul 28 18:04:02.755091 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.755044    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="01f44f4b-1f2b-4e6e-af48-6aa287b10d00" containerName="main"
Jul 28 18:04:02.757488 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.757471    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:04:02.761762 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.761739    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 28 18:04:02.761888 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.761791    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-5qx5m\""
Jul 28 18:04:02.761888 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.761850    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 28 18:04:02.761888 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.761877    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-g9jzg\""
Jul 28 18:04:02.762059 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.761927    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 28 18:04:02.774251 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.774228    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"]
Jul 28 18:04:02.785566 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.785541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4\" (UID: \"bb2d1f51-ba9a-4a40-a228-2bc6c465068c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:04:02.785686 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.785575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxt7\" (UniqueName: \"kubernetes.io/projected/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-kube-api-access-pfxt7\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4\" (UID: \"bb2d1f51-ba9a-4a40-a228-2bc6c465068c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:04:02.886094 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.886046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4\" (UID: \"bb2d1f51-ba9a-4a40-a228-2bc6c465068c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:04:02.886094 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.886095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxt7\" (UniqueName: \"kubernetes.io/projected/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-kube-api-access-pfxt7\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4\" (UID: \"bb2d1f51-ba9a-4a40-a228-2bc6c465068c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:04:02.888490 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.888467    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4\" (UID: \"bb2d1f51-ba9a-4a40-a228-2bc6c465068c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:04:02.895920 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:02.895893    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxt7\" (UniqueName: \"kubernetes.io/projected/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-kube-api-access-pfxt7\") pod \"stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4\" (UID: \"bb2d1f51-ba9a-4a40-a228-2bc6c465068c\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:04:03.067929 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:03.067897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:04:03.229553 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:03.229528    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"]
Jul 28 18:04:03.231230 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:04:03.231199    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2d1f51_ba9a_4a40_a228_2bc6c465068c.slice/crio-b2032016b17c24098e05b7d9ff3b4b24ed71a1c2de039dafec40421c0d4006ff WatchSource:0}: Error finding container b2032016b17c24098e05b7d9ff3b4b24ed71a1c2de039dafec40421c0d4006ff: Status 404 returned error can't find the container with id b2032016b17c24098e05b7d9ff3b4b24ed71a1c2de039dafec40421c0d4006ff
Jul 28 18:04:03.427283 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:03.427196    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4" event={"ID":"bb2d1f51-ba9a-4a40-a228-2bc6c465068c","Type":"ContainerStarted","Data":"418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194"}
Jul 28 18:04:03.427283 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:03.427260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4" event={"ID":"bb2d1f51-ba9a-4a40-a228-2bc6c465068c","Type":"ContainerStarted","Data":"b2032016b17c24098e05b7d9ff3b4b24ed71a1c2de039dafec40421c0d4006ff"}
Jul 28 18:04:03.427283 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:03.427281    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:04:03.449040 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:03.448989    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4" podStartSLOduration=1.448975916 podStartE2EDuration="1.448975916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:03.446016797 +0000 UTC m=+3231.675215900" watchObservedRunningTime="2026-07-28 18:04:03.448975916 +0000 UTC m=+3231.678175018"
Jul 28 18:04:34.434464 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:04:34.434432    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:05:12.498340 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:05:12.498306    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:05:12.508645 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:05:12.508623    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:06:10.816294 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:10.816260    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"]
Jul 28 18:06:14.434320 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:14.434285    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4" podUID="bb2d1f51-ba9a-4a40-a228-2bc6c465068c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:06:18.075463 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.075428    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"]
Jul 28 18:06:18.079305 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.079281    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.081949 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.081928    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"seaweedfs-s3-creds\""
Jul 28 18:06:18.092415 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.092393    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"]
Jul 28 18:06:18.174590 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.174559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-dshm\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.174745 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.174595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e320f9-dc34-415a-80de-245c6bf7c04f-tls-certs\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.174745 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.174633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-tmp-dir\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.174745 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.174658    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-kserve-provision-location\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.174745 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.174676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-home\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.174745 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.174734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v27lh\" (UniqueName: \"kubernetes.io/projected/f4e320f9-dc34-415a-80de-245c6bf7c04f-kube-api-access-v27lh\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.174916 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.174750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-model-cache\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276050 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-home\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276219 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v27lh\" (UniqueName: \"kubernetes.io/projected/f4e320f9-dc34-415a-80de-245c6bf7c04f-kube-api-access-v27lh\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276219 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-model-cache\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276219 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276156    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-dshm\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276219 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e320f9-dc34-415a-80de-245c6bf7c04f-tls-certs\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276219 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276211    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-tmp-dir\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276484 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-kserve-provision-location\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276484 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276466    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-home\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276598 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276493    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-model-cache\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276598 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276548    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-kserve-provision-location\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.276670 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.276628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-tmp-dir\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.278388 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.278366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-dshm\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.278568 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.278552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e320f9-dc34-415a-80de-245c6bf7c04f-tls-certs\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.285748 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.285726    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v27lh\" (UniqueName: \"kubernetes.io/projected/f4e320f9-dc34-415a-80de-245c6bf7c04f-kube-api-access-v27lh\") pod \"stop-feature-test-kserve-65499d9cfd-2sm6g\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.389634 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.389553    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:18.546329 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:18.546299    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"]
Jul 28 18:06:18.547595 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:06:18.547570    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e320f9_dc34_415a_80de_245c6bf7c04f.slice/crio-b0a848515222445f234645806314555c651ca519bd23510ec5a4fc803fd85937 WatchSource:0}: Error finding container b0a848515222445f234645806314555c651ca519bd23510ec5a4fc803fd85937: Status 404 returned error can't find the container with id b0a848515222445f234645806314555c651ca519bd23510ec5a4fc803fd85937
Jul 28 18:06:19.101611 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:19.101566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" event={"ID":"f4e320f9-dc34-415a-80de-245c6bf7c04f","Type":"ContainerStarted","Data":"08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11"}
Jul 28 18:06:19.102012 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:19.101626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" event={"ID":"f4e320f9-dc34-415a-80de-245c6bf7c04f","Type":"ContainerStarted","Data":"b0a848515222445f234645806314555c651ca519bd23510ec5a4fc803fd85937"}
Jul 28 18:06:25.836072 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:25.836012    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4" podUID="bb2d1f51-ba9a-4a40-a228-2bc6c465068c" containerName="main" containerID="cri-o://418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194" gracePeriod=45
Jul 28 18:06:26.094059 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.094001    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:06:26.138840 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.138806    2579 generic.go:356] "Generic (PLEG): container finished" podID="bb2d1f51-ba9a-4a40-a228-2bc6c465068c" containerID="418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194" exitCode=0
Jul 28 18:06:26.139017 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.138844    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"
Jul 28 18:06:26.139017 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.138887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4" event={"ID":"bb2d1f51-ba9a-4a40-a228-2bc6c465068c","Type":"ContainerDied","Data":"418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194"}
Jul 28 18:06:26.139017 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.138945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4" event={"ID":"bb2d1f51-ba9a-4a40-a228-2bc6c465068c","Type":"ContainerDied","Data":"b2032016b17c24098e05b7d9ff3b4b24ed71a1c2de039dafec40421c0d4006ff"}
Jul 28 18:06:26.139017 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.138980    2579 scope.go:117] "RemoveContainer" containerID="418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194"
Jul 28 18:06:26.140159 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.140073    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfxt7\" (UniqueName: \"kubernetes.io/projected/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-kube-api-access-pfxt7\") pod \"bb2d1f51-ba9a-4a40-a228-2bc6c465068c\" (UID: \"bb2d1f51-ba9a-4a40-a228-2bc6c465068c\") "
Jul 28 18:06:26.140241 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.140195    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-tls-certs\") pod \"bb2d1f51-ba9a-4a40-a228-2bc6c465068c\" (UID: \"bb2d1f51-ba9a-4a40-a228-2bc6c465068c\") "
Jul 28 18:06:26.142508 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.142482    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-kube-api-access-pfxt7" (OuterVolumeSpecName: "kube-api-access-pfxt7") pod "bb2d1f51-ba9a-4a40-a228-2bc6c465068c" (UID: "bb2d1f51-ba9a-4a40-a228-2bc6c465068c"). InnerVolumeSpecName "kube-api-access-pfxt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:06:26.142624 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.142472    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb2d1f51-ba9a-4a40-a228-2bc6c465068c" (UID: "bb2d1f51-ba9a-4a40-a228-2bc6c465068c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:06:26.156412 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.156379    2579 scope.go:117] "RemoveContainer" containerID="418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194"
Jul 28 18:06:26.156687 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:06:26.156669    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194\": container with ID starting with 418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194 not found: ID does not exist" containerID="418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194"
Jul 28 18:06:26.156750 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.156693    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194"} err="failed to get container status \"418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194\": rpc error: code = NotFound desc = could not find container \"418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194\": container with ID starting with 418f22ee14bca298866c445e417b7fce4774ff59bbf2b4be84d705bec6323194 not found: ID does not exist"
Jul 28 18:06:26.241469 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.241438    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:06:26.241469 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.241463    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfxt7\" (UniqueName: \"kubernetes.io/projected/bb2d1f51-ba9a-4a40-a228-2bc6c465068c-kube-api-access-pfxt7\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:06:26.458354 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.458277    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"]
Jul 28 18:06:26.465252 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:26.465227    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6ff9c88bb8-kwtf4"]
Jul 28 18:06:28.320585 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:28.320547    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2d1f51-ba9a-4a40-a228-2bc6c465068c" path="/var/lib/kubelet/pods/bb2d1f51-ba9a-4a40-a228-2bc6c465068c/volumes"
Jul 28 18:06:38.202436 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:38.202403    2579 generic.go:356] "Generic (PLEG): container finished" podID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerID="08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11" exitCode=0
Jul 28 18:06:38.202834 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:38.202476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" event={"ID":"f4e320f9-dc34-415a-80de-245c6bf7c04f","Type":"ContainerDied","Data":"08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11"}
Jul 28 18:06:39.209420 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:39.209385    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" event={"ID":"f4e320f9-dc34-415a-80de-245c6bf7c04f","Type":"ContainerStarted","Data":"8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438"}
Jul 28 18:06:39.239585 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:39.239529    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podStartSLOduration=1.881175377 podStartE2EDuration="21.2395098s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.358334423s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:06:39.237291001 +0000 UTC m=+3387.466490098" watchObservedRunningTime="2026-07-28 18:06:39.2395098 +0000 UTC m=+3387.468708906"
Jul 28 18:06:39.389680 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:39.389646    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:48.390513 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:48.390478    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:06:48.392097 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:48.392055    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:06:58.390938 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:06:58.390895    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:07:08.390392 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:07:08.390305    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:07:12.797410 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:07:12.797377    2579 scope.go:117] "RemoveContainer" containerID="575b45218d53d8d2cb64cfa5f337ecca23d28127d47d49a3d35fd1cb2a28562b"
Jul 28 18:07:18.390599 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:07:18.390554    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:07:28.390105 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:07:28.390039    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:07:38.390257 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:07:38.390198    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:07:48.389992 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:07:48.389928    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:07:58.390714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:07:58.390662    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:08:08.390491 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:08.390450    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:08:18.391039 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:18.390986    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:08:28.400610 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:28.400570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:08:28.409185 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:28.409155    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:08:29.788606 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:29.788572    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"]
Jul 28 18:08:44.821686 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:44.821575    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" containerID="cri-o://8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438" gracePeriod=45
Jul 28 18:08:45.402070 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.402025    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": dial tcp 10.132.0.79:8000: connect: connection refused"
Jul 28 18:08:45.578627 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.578600    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:08:45.674907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.674801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-kserve-provision-location\") pod \"f4e320f9-dc34-415a-80de-245c6bf7c04f\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") "
Jul 28 18:08:45.675126 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.674932    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-home\") pod \"f4e320f9-dc34-415a-80de-245c6bf7c04f\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") "
Jul 28 18:08:45.675126 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.674978    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e320f9-dc34-415a-80de-245c6bf7c04f-tls-certs\") pod \"f4e320f9-dc34-415a-80de-245c6bf7c04f\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") "
Jul 28 18:08:45.675126 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.675018    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v27lh\" (UniqueName: \"kubernetes.io/projected/f4e320f9-dc34-415a-80de-245c6bf7c04f-kube-api-access-v27lh\") pod \"f4e320f9-dc34-415a-80de-245c6bf7c04f\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") "
Jul 28 18:08:45.675126 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.675046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-model-cache\") pod \"f4e320f9-dc34-415a-80de-245c6bf7c04f\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") "
Jul 28 18:08:45.675126 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.675102    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-tmp-dir\") pod \"f4e320f9-dc34-415a-80de-245c6bf7c04f\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") "
Jul 28 18:08:45.675417 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.675166    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-dshm\") pod \"f4e320f9-dc34-415a-80de-245c6bf7c04f\" (UID: \"f4e320f9-dc34-415a-80de-245c6bf7c04f\") "
Jul 28 18:08:45.675417 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.675383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f4e320f9-dc34-415a-80de-245c6bf7c04f" (UID: "f4e320f9-dc34-415a-80de-245c6bf7c04f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:08:45.675583 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.675558    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:08:45.675769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.675738    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-home" (OuterVolumeSpecName: "home") pod "f4e320f9-dc34-415a-80de-245c6bf7c04f" (UID: "f4e320f9-dc34-415a-80de-245c6bf7c04f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:08:45.677576 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.677547    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-dshm" (OuterVolumeSpecName: "dshm") pod "f4e320f9-dc34-415a-80de-245c6bf7c04f" (UID: "f4e320f9-dc34-415a-80de-245c6bf7c04f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:08:45.677844 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.677824    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4e320f9-dc34-415a-80de-245c6bf7c04f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f4e320f9-dc34-415a-80de-245c6bf7c04f" (UID: "f4e320f9-dc34-415a-80de-245c6bf7c04f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:08:45.677918 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.677846    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4e320f9-dc34-415a-80de-245c6bf7c04f-kube-api-access-v27lh" (OuterVolumeSpecName: "kube-api-access-v27lh") pod "f4e320f9-dc34-415a-80de-245c6bf7c04f" (UID: "f4e320f9-dc34-415a-80de-245c6bf7c04f"). InnerVolumeSpecName "kube-api-access-v27lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:08:45.687588 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.687556    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f4e320f9-dc34-415a-80de-245c6bf7c04f" (UID: "f4e320f9-dc34-415a-80de-245c6bf7c04f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:08:45.741399 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.741360    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f4e320f9-dc34-415a-80de-245c6bf7c04f" (UID: "f4e320f9-dc34-415a-80de-245c6bf7c04f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:08:45.776284 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.776254    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:08:45.776284 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.776285    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:08:45.776447 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.776294    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4e320f9-dc34-415a-80de-245c6bf7c04f-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:08:45.776447 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.776307    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v27lh\" (UniqueName: \"kubernetes.io/projected/f4e320f9-dc34-415a-80de-245c6bf7c04f-kube-api-access-v27lh\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:08:45.776447 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.776315    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:08:45.776447 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.776322    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4e320f9-dc34-415a-80de-245c6bf7c04f-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:08:45.876242 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.876202    2579 generic.go:356] "Generic (PLEG): container finished" podID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerID="8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438" exitCode=0
Jul 28 18:08:45.876702 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.876262    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"
Jul 28 18:08:45.876702 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.876281    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" event={"ID":"f4e320f9-dc34-415a-80de-245c6bf7c04f","Type":"ContainerDied","Data":"8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438"}
Jul 28 18:08:45.876702 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.876332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g" event={"ID":"f4e320f9-dc34-415a-80de-245c6bf7c04f","Type":"ContainerDied","Data":"b0a848515222445f234645806314555c651ca519bd23510ec5a4fc803fd85937"}
Jul 28 18:08:45.876702 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.876354    2579 scope.go:117] "RemoveContainer" containerID="8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438"
Jul 28 18:08:45.886016 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.885996    2579 scope.go:117] "RemoveContainer" containerID="08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11"
Jul 28 18:08:45.894742 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.894725    2579 scope.go:117] "RemoveContainer" containerID="8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438"
Jul 28 18:08:45.895008 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:08:45.894987    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438\": container with ID starting with 8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438 not found: ID does not exist" containerID="8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438"
Jul 28 18:08:45.895175 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.895015    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438"} err="failed to get container status \"8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438\": rpc error: code = NotFound desc = could not find container \"8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438\": container with ID starting with 8158dadc42c4f5aae4f801a7612131989d21254ee983e7505c42733de825c438 not found: ID does not exist"
Jul 28 18:08:45.895175 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.895033    2579 scope.go:117] "RemoveContainer" containerID="08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11"
Jul 28 18:08:45.895337 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:08:45.895272    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11\": container with ID starting with 08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11 not found: ID does not exist" containerID="08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11"
Jul 28 18:08:45.895337 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.895293    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11"} err="failed to get container status \"08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11\": rpc error: code = NotFound desc = could not find container \"08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11\": container with ID starting with 08f7db3b605e72f6428e8d8881999c5a18c14340eb4f38508add871192dd0a11 not found: ID does not exist"
Jul 28 18:08:45.899975 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.899952    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"]
Jul 28 18:08:45.903177 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:45.903150    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-65499d9cfd-2sm6g"]
Jul 28 18:08:46.323940 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:08:46.323902    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" path="/var/lib/kubelet/pods/f4e320f9-dc34-415a-80de-245c6bf7c04f/volumes"
Jul 28 18:10:12.532660 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:10:12.532618    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:10:12.544361 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:10:12.544334    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:13:59.018218 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.018176    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"]
Jul 28 18:13:59.018649 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.018555    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="storage-initializer"
Jul 28 18:13:59.018649 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.018568    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="storage-initializer"
Jul 28 18:13:59.018649 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.018578    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main"
Jul 28 18:13:59.018649 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.018584    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main"
Jul 28 18:13:59.018649 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.018595    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2d1f51-ba9a-4a40-a228-2bc6c465068c" containerName="main"
Jul 28 18:13:59.018649 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.018601    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2d1f51-ba9a-4a40-a228-2bc6c465068c" containerName="main"
Jul 28 18:13:59.018846 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.018664    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2d1f51-ba9a-4a40-a228-2bc6c465068c" containerName="main"
Jul 28 18:13:59.018846 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.018672    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4e320f9-dc34-415a-80de-245c6bf7c04f" containerName="main"
Jul 28 18:13:59.021742 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.021721    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:13:59.024493 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.024444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Jul 28 18:13:59.024493 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.024444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Jul 28 18:13:59.024714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.024633    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-w6pm9\""
Jul 28 18:13:59.024714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.024691    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 28 18:13:59.024714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.024704    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-dzvkb\""
Jul 28 18:13:59.033604 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.033580    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"]
Jul 28 18:13:59.044425 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.044399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d8ad89-c989-45fa-b54e-7dddf6efa261-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-bw6wf\" (UID: \"54d8ad89-c989-45fa-b54e-7dddf6efa261\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:13:59.044542 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.044494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqbvm\" (UniqueName: \"kubernetes.io/projected/54d8ad89-c989-45fa-b54e-7dddf6efa261-kube-api-access-wqbvm\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-bw6wf\" (UID: \"54d8ad89-c989-45fa-b54e-7dddf6efa261\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:13:59.145036 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.144999    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqbvm\" (UniqueName: \"kubernetes.io/projected/54d8ad89-c989-45fa-b54e-7dddf6efa261-kube-api-access-wqbvm\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-bw6wf\" (UID: \"54d8ad89-c989-45fa-b54e-7dddf6efa261\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:13:59.145232 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.145098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d8ad89-c989-45fa-b54e-7dddf6efa261-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-bw6wf\" (UID: \"54d8ad89-c989-45fa-b54e-7dddf6efa261\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:13:59.147760 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.147724    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d8ad89-c989-45fa-b54e-7dddf6efa261-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-bw6wf\" (UID: \"54d8ad89-c989-45fa-b54e-7dddf6efa261\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:13:59.154724 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.154697    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqbvm\" (UniqueName: \"kubernetes.io/projected/54d8ad89-c989-45fa-b54e-7dddf6efa261-kube-api-access-wqbvm\") pod \"tls-verification-test-kserve-router-scheduler-5585754765-bw6wf\" (UID: \"54d8ad89-c989-45fa-b54e-7dddf6efa261\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:13:59.334144 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.334092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:13:59.466424 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:13:59.466340    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"]
Jul 28 18:13:59.468915 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:13:59.468887    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d8ad89_c989_45fa_b54e_7dddf6efa261.slice/crio-1762a483ccb40f1d4867d855a4ef546b5aeb695c88f41a9dc392e8eeab53e2de WatchSource:0}: Error finding container 1762a483ccb40f1d4867d855a4ef546b5aeb695c88f41a9dc392e8eeab53e2de: Status 404 returned error can't find the container with id 1762a483ccb40f1d4867d855a4ef546b5aeb695c88f41a9dc392e8eeab53e2de
Jul 28 18:14:00.391888 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:14:00.391841    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf" event={"ID":"54d8ad89-c989-45fa-b54e-7dddf6efa261","Type":"ContainerStarted","Data":"0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524"}
Jul 28 18:14:00.392312 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:14:00.391910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf" event={"ID":"54d8ad89-c989-45fa-b54e-7dddf6efa261","Type":"ContainerStarted","Data":"1762a483ccb40f1d4867d855a4ef546b5aeb695c88f41a9dc392e8eeab53e2de"}
Jul 28 18:14:00.392312 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:14:00.391939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:14:00.411951 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:14:00.411908    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf" podStartSLOduration=2.411895803 podStartE2EDuration="2.411895803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:14:00.410439264 +0000 UTC m=+3828.639638361" watchObservedRunningTime="2026-07-28 18:14:00.411895803 +0000 UTC m=+3828.641094898"
Jul 28 18:14:31.398528 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:14:31.398497    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:15:12.562677 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:15:12.562649    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:15:12.585610 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:15:12.585588    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:16:03.711057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:03.711015    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"]
Jul 28 18:16:11.398215 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:11.398173    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf" podUID="54d8ad89-c989-45fa-b54e-7dddf6efa261" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:16:18.731755 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:18.731686    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf" podUID="54d8ad89-c989-45fa-b54e-7dddf6efa261" containerName="main" containerID="cri-o://0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524" gracePeriod=45
Jul 28 18:16:18.993181 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:18.993122    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:16:19.043328 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.043299    2579 generic.go:356] "Generic (PLEG): container finished" podID="54d8ad89-c989-45fa-b54e-7dddf6efa261" containerID="0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524" exitCode=0
Jul 28 18:16:19.043496 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.043349    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"
Jul 28 18:16:19.043496 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.043383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf" event={"ID":"54d8ad89-c989-45fa-b54e-7dddf6efa261","Type":"ContainerDied","Data":"0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524"}
Jul 28 18:16:19.043496 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.043433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf" event={"ID":"54d8ad89-c989-45fa-b54e-7dddf6efa261","Type":"ContainerDied","Data":"1762a483ccb40f1d4867d855a4ef546b5aeb695c88f41a9dc392e8eeab53e2de"}
Jul 28 18:16:19.043496 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.043456    2579 scope.go:117] "RemoveContainer" containerID="0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524"
Jul 28 18:16:19.051713 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.051696    2579 scope.go:117] "RemoveContainer" containerID="0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524"
Jul 28 18:16:19.051973 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:16:19.051952    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524\": container with ID starting with 0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524 not found: ID does not exist" containerID="0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524"
Jul 28 18:16:19.052027 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.051982    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524"} err="failed to get container status \"0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524\": rpc error: code = NotFound desc = could not find container \"0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524\": container with ID starting with 0df70bf023d3fb2af6bb6c0f897f1d10dd589235263988102ab8d2c75b1a2524 not found: ID does not exist"
Jul 28 18:16:19.116116 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.116074    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqbvm\" (UniqueName: \"kubernetes.io/projected/54d8ad89-c989-45fa-b54e-7dddf6efa261-kube-api-access-wqbvm\") pod \"54d8ad89-c989-45fa-b54e-7dddf6efa261\" (UID: \"54d8ad89-c989-45fa-b54e-7dddf6efa261\") "
Jul 28 18:16:19.116257 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.116143    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d8ad89-c989-45fa-b54e-7dddf6efa261-tls-certs\") pod \"54d8ad89-c989-45fa-b54e-7dddf6efa261\" (UID: \"54d8ad89-c989-45fa-b54e-7dddf6efa261\") "
Jul 28 18:16:19.118328 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.118302    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d8ad89-c989-45fa-b54e-7dddf6efa261-kube-api-access-wqbvm" (OuterVolumeSpecName: "kube-api-access-wqbvm") pod "54d8ad89-c989-45fa-b54e-7dddf6efa261" (UID: "54d8ad89-c989-45fa-b54e-7dddf6efa261"). InnerVolumeSpecName "kube-api-access-wqbvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:16:19.118405 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.118384    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d8ad89-c989-45fa-b54e-7dddf6efa261-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "54d8ad89-c989-45fa-b54e-7dddf6efa261" (UID: "54d8ad89-c989-45fa-b54e-7dddf6efa261"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:16:19.216908 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.216874    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqbvm\" (UniqueName: \"kubernetes.io/projected/54d8ad89-c989-45fa-b54e-7dddf6efa261-kube-api-access-wqbvm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:16:19.216908 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.216905    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d8ad89-c989-45fa-b54e-7dddf6efa261-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:16:19.365711 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.365679    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"]
Jul 28 18:16:19.372092 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:19.372057    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-5585754765-bw6wf"]
Jul 28 18:16:20.319771 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:20.319734    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54d8ad89-c989-45fa-b54e-7dddf6efa261" path="/var/lib/kubelet/pods/54d8ad89-c989-45fa-b54e-7dddf6efa261/volumes"
Jul 28 18:16:36.685561 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.685528    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"]
Jul 28 18:16:36.687778 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.685891    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54d8ad89-c989-45fa-b54e-7dddf6efa261" containerName="main"
Jul 28 18:16:36.687778 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.685902    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="54d8ad89-c989-45fa-b54e-7dddf6efa261" containerName="main"
Jul 28 18:16:36.687778 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.686033    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="54d8ad89-c989-45fa-b54e-7dddf6efa261" containerName="main"
Jul 28 18:16:36.688812 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.688792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:16:36.692775 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.692751    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Jul 28 18:16:36.692902 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.692874    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 28 18:16:36.692902 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.692874    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Jul 28 18:16:36.693036 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.692876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-pqh2c\""
Jul 28 18:16:36.693248 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.693224    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-6slfp\""
Jul 28 18:16:36.701487 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.701464    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"]
Jul 28 18:16:36.785124 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.785070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258e9101-b744-4a77-b7e3-a38cf7f0f779-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm\" (UID: \"258e9101-b744-4a77-b7e3-a38cf7f0f779\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:16:36.785261 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.785188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkl4\" (UniqueName: \"kubernetes.io/projected/258e9101-b744-4a77-b7e3-a38cf7f0f779-kube-api-access-xzkl4\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm\" (UID: \"258e9101-b744-4a77-b7e3-a38cf7f0f779\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:16:36.886642 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.886611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258e9101-b744-4a77-b7e3-a38cf7f0f779-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm\" (UID: \"258e9101-b744-4a77-b7e3-a38cf7f0f779\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:16:36.886819 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.886658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkl4\" (UniqueName: \"kubernetes.io/projected/258e9101-b744-4a77-b7e3-a38cf7f0f779-kube-api-access-xzkl4\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm\" (UID: \"258e9101-b744-4a77-b7e3-a38cf7f0f779\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:16:36.889051 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.889022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258e9101-b744-4a77-b7e3-a38cf7f0f779-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm\" (UID: \"258e9101-b744-4a77-b7e3-a38cf7f0f779\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:16:36.895046 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.895022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkl4\" (UniqueName: \"kubernetes.io/projected/258e9101-b744-4a77-b7e3-a38cf7f0f779-kube-api-access-xzkl4\") pod \"prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm\" (UID: \"258e9101-b744-4a77-b7e3-a38cf7f0f779\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:16:36.999725 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:36.999649    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:16:37.135348 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:37.135308    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"]
Jul 28 18:16:37.139511 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:16:37.139477    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod258e9101_b744_4a77_b7e3_a38cf7f0f779.slice/crio-eb8a02526bfd750443cc7ac72343dbc0187e6ac5d53dfee02baea0ba77fff675 WatchSource:0}: Error finding container eb8a02526bfd750443cc7ac72343dbc0187e6ac5d53dfee02baea0ba77fff675: Status 404 returned error can't find the container with id eb8a02526bfd750443cc7ac72343dbc0187e6ac5d53dfee02baea0ba77fff675
Jul 28 18:16:38.139094 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:38.139039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm" event={"ID":"258e9101-b744-4a77-b7e3-a38cf7f0f779","Type":"ContainerStarted","Data":"d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec"}
Jul 28 18:16:38.139536 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:38.139123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm" event={"ID":"258e9101-b744-4a77-b7e3-a38cf7f0f779","Type":"ContainerStarted","Data":"eb8a02526bfd750443cc7ac72343dbc0187e6ac5d53dfee02baea0ba77fff675"}
Jul 28 18:16:38.139536 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:38.139153    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:16:38.159724 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:16:38.159675    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm" podStartSLOduration=2.159660462 podStartE2EDuration="2.159660462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:16:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:16:38.158534791 +0000 UTC m=+3986.387733888" watchObservedRunningTime="2026-07-28 18:16:38.159660462 +0000 UTC m=+3986.388859558"
Jul 28 18:17:09.145682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:17:09.145653    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:18:27.269109 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.269062    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"]
Jul 28 18:18:27.278620 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:18:27.278577    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:18:27.278620 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:18:27.278620 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:18:27.278620 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68" cmd=["/bin/sleep","15"]
Jul 28 18:18:27.278979 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.278643    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp" message="" err=<
Jul 28 18:18:27.278979 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:18:27.278979 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:18:27.278979 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 18:18:27.278979 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:18:27.278661    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:18:27.278979 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:18:27.278979 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:18:27.278979 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp" podUID="a886fdae-acc2-4dc1-b891-01b38bbf00e6" containerName="main" containerID="cri-o://95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68"
Jul 28 18:18:27.278979 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.278712    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp" podUID="a886fdae-acc2-4dc1-b891-01b38bbf00e6" containerName="main" containerID="cri-o://95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68" gracePeriod=60
Jul 28 18:18:27.336273 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.336231    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp" podUID="a886fdae-acc2-4dc1-b891-01b38bbf00e6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": dial tcp 10.132.0.77:8000: connect: connection refused"
Jul 28 18:18:27.642351 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.642323    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:18:27.662625 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.662500    2579 generic.go:356] "Generic (PLEG): container finished" podID="a886fdae-acc2-4dc1-b891-01b38bbf00e6" containerID="95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68" exitCode=0
Jul 28 18:18:27.662625 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.662558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp" event={"ID":"a886fdae-acc2-4dc1-b891-01b38bbf00e6","Type":"ContainerDied","Data":"95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68"}
Jul 28 18:18:27.662625 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.662609    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp" event={"ID":"a886fdae-acc2-4dc1-b891-01b38bbf00e6","Type":"ContainerDied","Data":"7330fde2da18bf4d956f4b8baca96f91166c07f4a12cc09ff7be1af9840bf24c"}
Jul 28 18:18:27.662972 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.662638    2579 scope.go:117] "RemoveContainer" containerID="95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68"
Jul 28 18:18:27.662972 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.662685    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"
Jul 28 18:18:27.671945 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.671923    2579 scope.go:117] "RemoveContainer" containerID="95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68"
Jul 28 18:18:27.672238 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:18:27.672217    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68\": container with ID starting with 95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68 not found: ID does not exist" containerID="95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68"
Jul 28 18:18:27.672298 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.672246    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68"} err="failed to get container status \"95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68\": rpc error: code = NotFound desc = could not find container \"95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68\": container with ID starting with 95988d61ad557f2ea7fd022f7cecb6671cae94660acd991fc6f3c615947c8c68 not found: ID does not exist"
Jul 28 18:18:27.686605 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.686578    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-home\") pod \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") "
Jul 28 18:18:27.686736 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.686612    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-dshm\") pod \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") "
Jul 28 18:18:27.686736 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.686668    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tls-certs\") pod \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") "
Jul 28 18:18:27.686736 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.686693    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tmp-dir\") pod \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") "
Jul 28 18:18:27.686736 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.686729    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-model-cache\") pod \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") "
Jul 28 18:18:27.686948 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.686763    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26rmg\" (UniqueName: \"kubernetes.io/projected/a886fdae-acc2-4dc1-b891-01b38bbf00e6-kube-api-access-26rmg\") pod \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\" (UID: \"a886fdae-acc2-4dc1-b891-01b38bbf00e6\") "
Jul 28 18:18:27.686948 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.686868    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-home" (OuterVolumeSpecName: "home") pod "a886fdae-acc2-4dc1-b891-01b38bbf00e6" (UID: "a886fdae-acc2-4dc1-b891-01b38bbf00e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:27.687102 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.687068    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:18:27.687277 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.687253    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "a886fdae-acc2-4dc1-b891-01b38bbf00e6" (UID: "a886fdae-acc2-4dc1-b891-01b38bbf00e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:27.687402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.687368    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a886fdae-acc2-4dc1-b891-01b38bbf00e6" (UID: "a886fdae-acc2-4dc1-b891-01b38bbf00e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:27.689248 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.689223    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a886fdae-acc2-4dc1-b891-01b38bbf00e6" (UID: "a886fdae-acc2-4dc1-b891-01b38bbf00e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:18:27.689354 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.689296    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a886fdae-acc2-4dc1-b891-01b38bbf00e6-kube-api-access-26rmg" (OuterVolumeSpecName: "kube-api-access-26rmg") pod "a886fdae-acc2-4dc1-b891-01b38bbf00e6" (UID: "a886fdae-acc2-4dc1-b891-01b38bbf00e6"). InnerVolumeSpecName "kube-api-access-26rmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:18:27.689435 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.689413    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-dshm" (OuterVolumeSpecName: "dshm") pod "a886fdae-acc2-4dc1-b891-01b38bbf00e6" (UID: "a886fdae-acc2-4dc1-b891-01b38bbf00e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:18:27.787522 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.787482    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:18:27.787522 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.787512    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:18:27.787522 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.787522    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:18:27.787522 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.787530    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a886fdae-acc2-4dc1-b891-01b38bbf00e6-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:18:27.787901 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.787539    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26rmg\" (UniqueName: \"kubernetes.io/projected/a886fdae-acc2-4dc1-b891-01b38bbf00e6-kube-api-access-26rmg\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:18:27.986874 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.986841    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"]
Jul 28 18:18:27.990525 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:27.990495    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-74rrp"]
Jul 28 18:18:28.320430 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:28.320386    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a886fdae-acc2-4dc1-b891-01b38bbf00e6" path="/var/lib/kubelet/pods/a886fdae-acc2-4dc1-b891-01b38bbf00e6/volumes"
Jul 28 18:18:39.424616 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.424581    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"]
Jul 28 18:18:39.425105 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.424957    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a886fdae-acc2-4dc1-b891-01b38bbf00e6" containerName="main"
Jul 28 18:18:39.425105 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.424968    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a886fdae-acc2-4dc1-b891-01b38bbf00e6" containerName="main"
Jul 28 18:18:39.425105 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.425038    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a886fdae-acc2-4dc1-b891-01b38bbf00e6" containerName="main"
Jul 28 18:18:39.428148 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.428130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.431389 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.431356    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"seaweedfs-s3-creds\""
Jul 28 18:18:39.440587 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.440560    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"]
Jul 28 18:18:39.495385 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.495357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.495532 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.495391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgp7\" (UniqueName: \"kubernetes.io/projected/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kube-api-access-9bgp7\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.495532 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.495414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.495532 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.495494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.495668 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.495532    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.495668 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.495631    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.495668 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.495664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597067 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597037    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597232 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgp7\" (UniqueName: \"kubernetes.io/projected/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kube-api-access-9bgp7\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597232 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597232 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597232 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597159    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597232 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597232 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597215    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597565 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tmp-dir\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597565 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kserve-provision-location\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597681 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597586    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-home\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.597681 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.597648    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-model-cache\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.599842 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.599815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-dshm\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.600032 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.600016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tls-certs\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.605626 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.605602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgp7\" (UniqueName: \"kubernetes.io/projected/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kube-api-access-9bgp7\") pod \"prestop-hook-test-kserve-68f6bd7b79-mg4fw\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.739186 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.739111    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:39.880822 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:39.880782    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"]
Jul 28 18:18:39.884971 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:18:39.884945    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66c1f47_9a0a_4539_8995_5f5d43e31f91.slice/crio-10d61b17ad12420139f391f04bc48058a7cab86f1d69b04f1e94dc70e72efe1f WatchSource:0}: Error finding container 10d61b17ad12420139f391f04bc48058a7cab86f1d69b04f1e94dc70e72efe1f: Status 404 returned error can't find the container with id 10d61b17ad12420139f391f04bc48058a7cab86f1d69b04f1e94dc70e72efe1f
Jul 28 18:18:40.726226 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:40.726188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw" event={"ID":"e66c1f47-9a0a-4539-8995-5f5d43e31f91","Type":"ContainerStarted","Data":"971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21"}
Jul 28 18:18:40.726653 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:40.726268    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw" event={"ID":"e66c1f47-9a0a-4539-8995-5f5d43e31f91","Type":"ContainerStarted","Data":"10d61b17ad12420139f391f04bc48058a7cab86f1d69b04f1e94dc70e72efe1f"}
Jul 28 18:18:56.807822 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:56.807784    2579 generic.go:356] "Generic (PLEG): container finished" podID="e66c1f47-9a0a-4539-8995-5f5d43e31f91" containerID="971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21" exitCode=0
Jul 28 18:18:56.808330 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:56.807846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw" event={"ID":"e66c1f47-9a0a-4539-8995-5f5d43e31f91","Type":"ContainerDied","Data":"971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21"}
Jul 28 18:18:57.815813 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:57.815778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw" event={"ID":"e66c1f47-9a0a-4539-8995-5f5d43e31f91","Type":"ContainerStarted","Data":"e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473"}
Jul 28 18:18:57.840458 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:57.840399    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw" podStartSLOduration=2.6558133269999997 podStartE2EDuration="18.840383984s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.184570657s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:18:57.837616924 +0000 UTC m=+4126.066816035" watchObservedRunningTime="2026-07-28 18:18:57.840383984 +0000 UTC m=+4126.069583079"
Jul 28 18:18:58.740220 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:58.740190    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:59.740318 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:59.740269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:18:59.741910 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:18:59.741876    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw" podUID="e66c1f47-9a0a-4539-8995-5f5d43e31f91" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": dial tcp 10.132.0.82:8000: connect: connection refused"
Jul 28 18:19:04.907242 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:04.907159    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"]
Jul 28 18:19:04.910009 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:04.909983    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:19:04.914525 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:04.914491    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-epp-sa-dockercfg-4c7kz\""
Jul 28 18:19:04.914696 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:04.914541    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-nn5fq\""
Jul 28 18:19:04.914781 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:04.914557    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 28 18:19:04.914781 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:04.914598    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 28 18:19:04.914946 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:04.914646    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 28 18:19:04.924633 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:04.924608    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"]
Jul 28 18:19:04.938760 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:04.938729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd7x9\" (UniqueName: \"kubernetes.io/projected/2095e6c0-9ff5-4332-a02d-7eb973625488-kube-api-access-kd7x9\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj\" (UID: \"2095e6c0-9ff5-4332-a02d-7eb973625488\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:19:04.938905 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:04.938780    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2095e6c0-9ff5-4332-a02d-7eb973625488-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj\" (UID: \"2095e6c0-9ff5-4332-a02d-7eb973625488\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:19:05.039606 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:05.039567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd7x9\" (UniqueName: \"kubernetes.io/projected/2095e6c0-9ff5-4332-a02d-7eb973625488-kube-api-access-kd7x9\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj\" (UID: \"2095e6c0-9ff5-4332-a02d-7eb973625488\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:19:05.039762 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:05.039630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2095e6c0-9ff5-4332-a02d-7eb973625488-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj\" (UID: \"2095e6c0-9ff5-4332-a02d-7eb973625488\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:19:05.042109 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:05.042067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2095e6c0-9ff5-4332-a02d-7eb973625488-tls-certs\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj\" (UID: \"2095e6c0-9ff5-4332-a02d-7eb973625488\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:19:05.049692 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:05.049670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd7x9\" (UniqueName: \"kubernetes.io/projected/2095e6c0-9ff5-4332-a02d-7eb973625488-kube-api-access-kd7x9\") pod \"autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj\" (UID: \"2095e6c0-9ff5-4332-a02d-7eb973625488\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:19:05.224634 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:05.224547    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:19:05.364273 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:05.364246    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"]
Jul 28 18:19:05.366379 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:19:05.366354    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2095e6c0_9ff5_4332_a02d_7eb973625488.slice/crio-6f941384108bde674cbcda33c0309b41bcb316d3790e0c66c0984908c2178974 WatchSource:0}: Error finding container 6f941384108bde674cbcda33c0309b41bcb316d3790e0c66c0984908c2178974: Status 404 returned error can't find the container with id 6f941384108bde674cbcda33c0309b41bcb316d3790e0c66c0984908c2178974
Jul 28 18:19:05.859783 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:05.859743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj" event={"ID":"2095e6c0-9ff5-4332-a02d-7eb973625488","Type":"ContainerStarted","Data":"2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30"}
Jul 28 18:19:05.859961 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:05.859811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj" event={"ID":"2095e6c0-9ff5-4332-a02d-7eb973625488","Type":"ContainerStarted","Data":"6f941384108bde674cbcda33c0309b41bcb316d3790e0c66c0984908c2178974"}
Jul 28 18:19:05.859961 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:05.859886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:19:05.879469 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:05.879414    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj" podStartSLOduration=1.879399671 podStartE2EDuration="1.879399671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:19:05.87765377 +0000 UTC m=+4134.106852879" watchObservedRunningTime="2026-07-28 18:19:05.879399671 +0000 UTC m=+4134.108598766"
Jul 28 18:19:09.363749 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:09.363711    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"]
Jul 28 18:19:09.368537 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:09.368508    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"]
Jul 28 18:19:19.146365 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:19.146318    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm" podUID="258e9101-b744-4a77-b7e3-a38cf7f0f779" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:19:24.385019 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.384970    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm" podUID="258e9101-b744-4a77-b7e3-a38cf7f0f779" containerName="main" containerID="cri-o://d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec" gracePeriod=45
Jul 28 18:19:24.392523 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.392488    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw" podUID="e66c1f47-9a0a-4539-8995-5f5d43e31f91" containerName="main" containerID="cri-o://e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473" gracePeriod=45
Jul 28 18:19:24.643090 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.643020    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:19:24.725265 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.725232    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258e9101-b744-4a77-b7e3-a38cf7f0f779-tls-certs\") pod \"258e9101-b744-4a77-b7e3-a38cf7f0f779\" (UID: \"258e9101-b744-4a77-b7e3-a38cf7f0f779\") "
Jul 28 18:19:24.725441 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.725322    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzkl4\" (UniqueName: \"kubernetes.io/projected/258e9101-b744-4a77-b7e3-a38cf7f0f779-kube-api-access-xzkl4\") pod \"258e9101-b744-4a77-b7e3-a38cf7f0f779\" (UID: \"258e9101-b744-4a77-b7e3-a38cf7f0f779\") "
Jul 28 18:19:24.727553 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.727517    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/258e9101-b744-4a77-b7e3-a38cf7f0f779-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "258e9101-b744-4a77-b7e3-a38cf7f0f779" (UID: "258e9101-b744-4a77-b7e3-a38cf7f0f779"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:19:24.727681 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.727603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/258e9101-b744-4a77-b7e3-a38cf7f0f779-kube-api-access-xzkl4" (OuterVolumeSpecName: "kube-api-access-xzkl4") pod "258e9101-b744-4a77-b7e3-a38cf7f0f779" (UID: "258e9101-b744-4a77-b7e3-a38cf7f0f779"). InnerVolumeSpecName "kube-api-access-xzkl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:19:24.826658 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.826625    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/258e9101-b744-4a77-b7e3-a38cf7f0f779-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:19:24.826658 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.826654    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzkl4\" (UniqueName: \"kubernetes.io/projected/258e9101-b744-4a77-b7e3-a38cf7f0f779-kube-api-access-xzkl4\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:19:24.961371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.961275    2579 generic.go:356] "Generic (PLEG): container finished" podID="258e9101-b744-4a77-b7e3-a38cf7f0f779" containerID="d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec" exitCode=0
Jul 28 18:19:24.961371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.961313    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"
Jul 28 18:19:24.961371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.961350    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm" event={"ID":"258e9101-b744-4a77-b7e3-a38cf7f0f779","Type":"ContainerDied","Data":"d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec"}
Jul 28 18:19:24.961657 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.961400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm" event={"ID":"258e9101-b744-4a77-b7e3-a38cf7f0f779","Type":"ContainerDied","Data":"eb8a02526bfd750443cc7ac72343dbc0187e6ac5d53dfee02baea0ba77fff675"}
Jul 28 18:19:24.961657 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.961422    2579 scope.go:117] "RemoveContainer" containerID="d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec"
Jul 28 18:19:24.972298 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.972280    2579 scope.go:117] "RemoveContainer" containerID="d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec"
Jul 28 18:19:24.972546 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:19:24.972526    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec\": container with ID starting with d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec not found: ID does not exist" containerID="d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec"
Jul 28 18:19:24.972609 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.972553    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec"} err="failed to get container status \"d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec\": rpc error: code = NotFound desc = could not find container \"d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec\": container with ID starting with d70aa775d82dbfddbbb1315ca69cf23d66143cb2925f1e7ef23b34d81aa053ec not found: ID does not exist"
Jul 28 18:19:24.984972 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.984946    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"]
Jul 28 18:19:24.990625 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:24.990605    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-6dc6dff85-lvslm"]
Jul 28 18:19:26.320421 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:26.320386    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="258e9101-b744-4a77-b7e3-a38cf7f0f779" path="/var/lib/kubelet/pods/258e9101-b744-4a77-b7e3-a38cf7f0f779/volumes"
Jul 28 18:19:36.865931 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:19:36.865898    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:20:10.032464 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.032434    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-68f6bd7b79-mg4fw_e66c1f47-9a0a-4539-8995-5f5d43e31f91/main/0.log"
Jul 28 18:20:10.034432 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.034413    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:20:10.164682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.164595    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tmp-dir\") pod \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") "
Jul 28 18:20:10.164682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.164644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bgp7\" (UniqueName: \"kubernetes.io/projected/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kube-api-access-9bgp7\") pod \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") "
Jul 28 18:20:10.164924 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.164690    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tls-certs\") pod \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") "
Jul 28 18:20:10.164924 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.164781    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-dshm\") pod \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") "
Jul 28 18:20:10.164924 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.164809    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-model-cache\") pod \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") "
Jul 28 18:20:10.165107 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.164987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-home\") pod \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") "
Jul 28 18:20:10.165107 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.165037    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kserve-provision-location\") pod \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\" (UID: \"e66c1f47-9a0a-4539-8995-5f5d43e31f91\") "
Jul 28 18:20:10.165107 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.165062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-model-cache" (OuterVolumeSpecName: "model-cache") pod "e66c1f47-9a0a-4539-8995-5f5d43e31f91" (UID: "e66c1f47-9a0a-4539-8995-5f5d43e31f91"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:20:10.165808 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.165770    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-home" (OuterVolumeSpecName: "home") pod "e66c1f47-9a0a-4539-8995-5f5d43e31f91" (UID: "e66c1f47-9a0a-4539-8995-5f5d43e31f91"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:20:10.166682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.166647    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e66c1f47-9a0a-4539-8995-5f5d43e31f91" (UID: "e66c1f47-9a0a-4539-8995-5f5d43e31f91"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:20:10.166682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.166675    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:20:10.166941 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.166710    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:20:10.167306 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.167273    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kube-api-access-9bgp7" (OuterVolumeSpecName: "kube-api-access-9bgp7") pod "e66c1f47-9a0a-4539-8995-5f5d43e31f91" (UID: "e66c1f47-9a0a-4539-8995-5f5d43e31f91"). InnerVolumeSpecName "kube-api-access-9bgp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:20:10.167439 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.167308    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-dshm" (OuterVolumeSpecName: "dshm") pod "e66c1f47-9a0a-4539-8995-5f5d43e31f91" (UID: "e66c1f47-9a0a-4539-8995-5f5d43e31f91"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:20:10.175660 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.175584    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e66c1f47-9a0a-4539-8995-5f5d43e31f91" (UID: "e66c1f47-9a0a-4539-8995-5f5d43e31f91"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:20:10.203061 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.203036    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-prestop-hook-22727e61_prestop-hook-test-kserve-68f6bd7b79-mg4fw_e66c1f47-9a0a-4539-8995-5f5d43e31f91/main/0.log"
Jul 28 18:20:10.203447 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.203424    2579 generic.go:356] "Generic (PLEG): container finished" podID="e66c1f47-9a0a-4539-8995-5f5d43e31f91" containerID="e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473" exitCode=137
Jul 28 18:20:10.203534 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.203482    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw" event={"ID":"e66c1f47-9a0a-4539-8995-5f5d43e31f91","Type":"ContainerDied","Data":"e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473"}
Jul 28 18:20:10.203534 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.203488    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"
Jul 28 18:20:10.203610 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.203557    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw" event={"ID":"e66c1f47-9a0a-4539-8995-5f5d43e31f91","Type":"ContainerDied","Data":"10d61b17ad12420139f391f04bc48058a7cab86f1d69b04f1e94dc70e72efe1f"}
Jul 28 18:20:10.203610 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.203582    2579 scope.go:117] "RemoveContainer" containerID="e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473"
Jul 28 18:20:10.212013 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.211996    2579 scope.go:117] "RemoveContainer" containerID="971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21"
Jul 28 18:20:10.219876 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.219857    2579 scope.go:117] "RemoveContainer" containerID="e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473"
Jul 28 18:20:10.220168 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:20:10.220146    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473\": container with ID starting with e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473 not found: ID does not exist" containerID="e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473"
Jul 28 18:20:10.220269 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.220178    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473"} err="failed to get container status \"e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473\": rpc error: code = NotFound desc = could not find container \"e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473\": container with ID starting with e6ae964319c10b4e52ec2756ce1c9c0d48486cecde39e70e5bd236b708195473 not found: ID does not exist"
Jul 28 18:20:10.220269 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.220197    2579 scope.go:117] "RemoveContainer" containerID="971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21"
Jul 28 18:20:10.220448 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:20:10.220431    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21\": container with ID starting with 971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21 not found: ID does not exist" containerID="971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21"
Jul 28 18:20:10.220539 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.220455    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21"} err="failed to get container status \"971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21\": rpc error: code = NotFound desc = could not find container \"971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21\": container with ID starting with 971cae59f93f8630d67764983f64b59ed5b814b3c5959fca51a7f706348add21 not found: ID does not exist"
Jul 28 18:20:10.240142 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.240112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e66c1f47-9a0a-4539-8995-5f5d43e31f91" (UID: "e66c1f47-9a0a-4539-8995-5f5d43e31f91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:20:10.268013 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.267986    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:20:10.268013 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.268009    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:20:10.268166 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.268018    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:20:10.268166 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.268027    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e66c1f47-9a0a-4539-8995-5f5d43e31f91-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:20:10.268166 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.268039    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bgp7\" (UniqueName: \"kubernetes.io/projected/e66c1f47-9a0a-4539-8995-5f5d43e31f91-kube-api-access-9bgp7\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:20:10.521483 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.521400    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"]
Jul 28 18:20:10.524707 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:10.524680    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-68f6bd7b79-mg4fw"]
Jul 28 18:20:12.319830 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:12.319801    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e66c1f47-9a0a-4539-8995-5f5d43e31f91" path="/var/lib/kubelet/pods/e66c1f47-9a0a-4539-8995-5f5d43e31f91/volumes"
Jul 28 18:20:12.593058 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:12.592985    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:20:12.616519 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:12.616494    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:20:17.656281 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.656213    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"]
Jul 28 18:20:17.656759 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.656590    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="258e9101-b744-4a77-b7e3-a38cf7f0f779" containerName="main"
Jul 28 18:20:17.656759 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.656601    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="258e9101-b744-4a77-b7e3-a38cf7f0f779" containerName="main"
Jul 28 18:20:17.656759 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.656621    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66c1f47-9a0a-4539-8995-5f5d43e31f91" containerName="main"
Jul 28 18:20:17.656759 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.656628    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66c1f47-9a0a-4539-8995-5f5d43e31f91" containerName="main"
Jul 28 18:20:17.656759 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.656637    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e66c1f47-9a0a-4539-8995-5f5d43e31f91" containerName="storage-initializer"
Jul 28 18:20:17.656759 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.656644    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e66c1f47-9a0a-4539-8995-5f5d43e31f91" containerName="storage-initializer"
Jul 28 18:20:17.656759 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.656708    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="258e9101-b744-4a77-b7e3-a38cf7f0f779" containerName="main"
Jul 28 18:20:17.656759 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.656714    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e66c1f47-9a0a-4539-8995-5f5d43e31f91" containerName="main"
Jul 28 18:20:17.660819 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.660798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.663739 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.663713    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-mg22h\""
Jul 28 18:20:17.665286 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.665263    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Jul 28 18:20:17.665356 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.665290    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Jul 28 18:20:17.665356 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.665296    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 28 18:20:17.671306 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.671281    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"]
Jul 28 18:20:17.844943 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.844913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.844943 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.844948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.845212 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.844981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msc5\" (UniqueName: \"kubernetes.io/projected/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-kube-api-access-9msc5\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.845212 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.845050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.845212 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.845104    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.845212 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.845159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.910826 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.910751    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"]
Jul 28 18:20:17.913386 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.913370    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:20:17.916876 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.916855    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-hr4qb\""
Jul 28 18:20:17.923312 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.923289    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"]
Jul 28 18:20:17.946087 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.946061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.946207 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.946108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.946207 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.946151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msc5\" (UniqueName: \"kubernetes.io/projected/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-kube-api-access-9msc5\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.946207 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.946179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.946339 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.946302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.946411 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.946394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.946638 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.946606    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-model-cache\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.946727 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.946638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-home\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.946727 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.946683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tmp-dir\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.948336 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.948314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-dshm\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.948587 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.948571    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tls-certs\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.954801 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.954782    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msc5\" (UniqueName: \"kubernetes.io/projected/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-kube-api-access-9msc5\") pod \"rolling-upgrade-test-kserve-74b79d7f77-rsvrw\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:17.972306 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:17.972280    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:18.047560 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.047527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094794df-2957-4248-86bc-7d78a9bfca97-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd\" (UID: \"094794df-2957-4248-86bc-7d78a9bfca97\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:20:18.047694 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.047633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78p8\" (UniqueName: \"kubernetes.io/projected/094794df-2957-4248-86bc-7d78a9bfca97-kube-api-access-d78p8\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd\" (UID: \"094794df-2957-4248-86bc-7d78a9bfca97\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:20:18.105706 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.105681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"]
Jul 28 18:20:18.108146 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:20:18.108119    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f815cb3_bd99_4d9c_ac10_47af5bc052ac.slice/crio-75164d07257ff8bb72a91367ea4b40f11b15201d8eed3c1cd7c0878f91dd5c4a WatchSource:0}: Error finding container 75164d07257ff8bb72a91367ea4b40f11b15201d8eed3c1cd7c0878f91dd5c4a: Status 404 returned error can't find the container with id 75164d07257ff8bb72a91367ea4b40f11b15201d8eed3c1cd7c0878f91dd5c4a
Jul 28 18:20:18.149088 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.149049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78p8\" (UniqueName: \"kubernetes.io/projected/094794df-2957-4248-86bc-7d78a9bfca97-kube-api-access-d78p8\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd\" (UID: \"094794df-2957-4248-86bc-7d78a9bfca97\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:20:18.149213 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.149184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094794df-2957-4248-86bc-7d78a9bfca97-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd\" (UID: \"094794df-2957-4248-86bc-7d78a9bfca97\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:20:18.151655 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.151634    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094794df-2957-4248-86bc-7d78a9bfca97-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd\" (UID: \"094794df-2957-4248-86bc-7d78a9bfca97\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:20:18.157505 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.157479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78p8\" (UniqueName: \"kubernetes.io/projected/094794df-2957-4248-86bc-7d78a9bfca97-kube-api-access-d78p8\") pod \"rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd\" (UID: \"094794df-2957-4248-86bc-7d78a9bfca97\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:20:18.224261 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.224223    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:20:18.244379 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.244343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw" event={"ID":"8f815cb3-bd99-4d9c-ac10-47af5bc052ac","Type":"ContainerStarted","Data":"bcc4857a98e85123f1883f41f4bcba55130bf6d6db79b7ec2405f3edf2120eec"}
Jul 28 18:20:18.244504 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.244408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw" event={"ID":"8f815cb3-bd99-4d9c-ac10-47af5bc052ac","Type":"ContainerStarted","Data":"75164d07257ff8bb72a91367ea4b40f11b15201d8eed3c1cd7c0878f91dd5c4a"}
Jul 28 18:20:18.264851 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.264790    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw" podStartSLOduration=1.264768162 podStartE2EDuration="1.264768162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:20:18.262985113 +0000 UTC m=+4206.492184231" watchObservedRunningTime="2026-07-28 18:20:18.264768162 +0000 UTC m=+4206.493967259"
Jul 28 18:20:18.357858 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.357834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"]
Jul 28 18:20:18.360406 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:20:18.360384    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094794df_2957_4248_86bc_7d78a9bfca97.slice/crio-d6f7ad352eb57193c74c711ce74fdc910a18f031303df0153b9ed4bf44f360c2 WatchSource:0}: Error finding container d6f7ad352eb57193c74c711ce74fdc910a18f031303df0153b9ed4bf44f360c2: Status 404 returned error can't find the container with id d6f7ad352eb57193c74c711ce74fdc910a18f031303df0153b9ed4bf44f360c2
Jul 28 18:20:18.973121 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:18.973062    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:19.251182 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:19.251089    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd" event={"ID":"094794df-2957-4248-86bc-7d78a9bfca97","Type":"ContainerStarted","Data":"937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b"}
Jul 28 18:20:19.251182 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:19.251153    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd" event={"ID":"094794df-2957-4248-86bc-7d78a9bfca97","Type":"ContainerStarted","Data":"d6f7ad352eb57193c74c711ce74fdc910a18f031303df0153b9ed4bf44f360c2"}
Jul 28 18:20:19.271718 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:19.271658    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd" podStartSLOduration=2.271639466 podStartE2EDuration="2.271639466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:20:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:20:19.270988784 +0000 UTC m=+4207.500187881" watchObservedRunningTime="2026-07-28 18:20:19.271639466 +0000 UTC m=+4207.500838563"
Jul 28 18:20:20.256137 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:20.256100    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:20:27.973147 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:27.973108    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:27.986251 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:27.986224    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:28.308263 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:28.308179    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:20:51.262134 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:20:51.262034    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:21:03.203749 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.203716    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"]
Jul 28 18:21:03.213754 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:21:03.213714    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:21:03.213754 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:21:03.213754 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:21:03.213754 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="bcc4857a98e85123f1883f41f4bcba55130bf6d6db79b7ec2405f3edf2120eec" cmd=["/bin/sleep","15"]
Jul 28 18:21:03.214010 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.213768    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw" message="" err=<
Jul 28 18:21:03.214010 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:21:03.214010 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:21:03.214010 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 18:21:03.214010 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:21:03.213781    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:21:03.214010 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:21:03.214010 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:21:03.214010 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw" podUID="8f815cb3-bd99-4d9c-ac10-47af5bc052ac" containerName="main" containerID="cri-o://bcc4857a98e85123f1883f41f4bcba55130bf6d6db79b7ec2405f3edf2120eec"
Jul 28 18:21:03.214010 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.213820    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw" podUID="8f815cb3-bd99-4d9c-ac10-47af5bc052ac" containerName="main" containerID="cri-o://bcc4857a98e85123f1883f41f4bcba55130bf6d6db79b7ec2405f3edf2120eec" gracePeriod=60
Jul 28 18:21:03.298365 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.298324    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw" podUID="8f815cb3-bd99-4d9c-ac10-47af5bc052ac" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": dial tcp 10.132.0.84:8000: connect: connection refused"
Jul 28 18:21:03.468763 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.468666    2579 generic.go:356] "Generic (PLEG): container finished" podID="8f815cb3-bd99-4d9c-ac10-47af5bc052ac" containerID="bcc4857a98e85123f1883f41f4bcba55130bf6d6db79b7ec2405f3edf2120eec" exitCode=0
Jul 28 18:21:03.468763 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.468709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw" event={"ID":"8f815cb3-bd99-4d9c-ac10-47af5bc052ac","Type":"ContainerDied","Data":"bcc4857a98e85123f1883f41f4bcba55130bf6d6db79b7ec2405f3edf2120eec"}
Jul 28 18:21:03.568255 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.568230    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:21:03.667275 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-dshm\") pod \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") "
Jul 28 18:21:03.667473 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667327    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tls-certs\") pod \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") "
Jul 28 18:21:03.667473 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667373    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-model-cache\") pod \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") "
Jul 28 18:21:03.667473 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667398    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9msc5\" (UniqueName: \"kubernetes.io/projected/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-kube-api-access-9msc5\") pod \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") "
Jul 28 18:21:03.667473 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667419    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-home\") pod \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") "
Jul 28 18:21:03.667473 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667462    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tmp-dir\") pod \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\" (UID: \"8f815cb3-bd99-4d9c-ac10-47af5bc052ac\") "
Jul 28 18:21:03.667760 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667672    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f815cb3-bd99-4d9c-ac10-47af5bc052ac" (UID: "8f815cb3-bd99-4d9c-ac10-47af5bc052ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:21:03.667760 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667712    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-home" (OuterVolumeSpecName: "home") pod "8f815cb3-bd99-4d9c-ac10-47af5bc052ac" (UID: "8f815cb3-bd99-4d9c-ac10-47af5bc052ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:21:03.667856 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667782    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:21:03.667856 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667799    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:21:03.667856 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.667835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f815cb3-bd99-4d9c-ac10-47af5bc052ac" (UID: "8f815cb3-bd99-4d9c-ac10-47af5bc052ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:21:03.669503 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.669474    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-dshm" (OuterVolumeSpecName: "dshm") pod "8f815cb3-bd99-4d9c-ac10-47af5bc052ac" (UID: "8f815cb3-bd99-4d9c-ac10-47af5bc052ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:21:03.669629 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.669537    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f815cb3-bd99-4d9c-ac10-47af5bc052ac" (UID: "8f815cb3-bd99-4d9c-ac10-47af5bc052ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:21:03.669629 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.669556    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-kube-api-access-9msc5" (OuterVolumeSpecName: "kube-api-access-9msc5") pod "8f815cb3-bd99-4d9c-ac10-47af5bc052ac" (UID: "8f815cb3-bd99-4d9c-ac10-47af5bc052ac"). InnerVolumeSpecName "kube-api-access-9msc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:21:03.769092 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.768995    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:21:03.769092 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.769026    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:21:03.769092 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.769036    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msc5\" (UniqueName: \"kubernetes.io/projected/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-kube-api-access-9msc5\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:21:03.769092 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:03.769044    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f815cb3-bd99-4d9c-ac10-47af5bc052ac-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:21:04.475398 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:04.475364    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"
Jul 28 18:21:04.475398 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:04.475392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw" event={"ID":"8f815cb3-bd99-4d9c-ac10-47af5bc052ac","Type":"ContainerDied","Data":"75164d07257ff8bb72a91367ea4b40f11b15201d8eed3c1cd7c0878f91dd5c4a"}
Jul 28 18:21:04.475883 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:04.475456    2579 scope.go:117] "RemoveContainer" containerID="bcc4857a98e85123f1883f41f4bcba55130bf6d6db79b7ec2405f3edf2120eec"
Jul 28 18:21:04.493654 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:04.493631    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"]
Jul 28 18:21:04.497175 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:04.497154    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-74b79d7f77-rsvrw"]
Jul 28 18:21:06.319961 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:06.319925    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f815cb3-bd99-4d9c-ac10-47af5bc052ac" path="/var/lib/kubelet/pods/8f815cb3-bd99-4d9c-ac10-47af5bc052ac/volumes"
Jul 28 18:21:17.277168 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:17.277122    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"]
Jul 28 18:21:21.262070 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:21.262021    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd" podUID="094794df-2957-4248-86bc-7d78a9bfca97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:21:31.261965 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:31.261928    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd" podUID="094794df-2957-4248-86bc-7d78a9bfca97" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:21:32.297280 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.297237    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd" podUID="094794df-2957-4248-86bc-7d78a9bfca97" containerName="main" containerID="cri-o://937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b" gracePeriod=45
Jul 28 18:21:32.552604 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.552548    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:21:32.621047 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.621017    2579 generic.go:356] "Generic (PLEG): container finished" podID="094794df-2957-4248-86bc-7d78a9bfca97" containerID="937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b" exitCode=0
Jul 28 18:21:32.621236 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.621062    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"
Jul 28 18:21:32.621236 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.621106    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd" event={"ID":"094794df-2957-4248-86bc-7d78a9bfca97","Type":"ContainerDied","Data":"937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b"}
Jul 28 18:21:32.621236 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.621159    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd" event={"ID":"094794df-2957-4248-86bc-7d78a9bfca97","Type":"ContainerDied","Data":"d6f7ad352eb57193c74c711ce74fdc910a18f031303df0153b9ed4bf44f360c2"}
Jul 28 18:21:32.621236 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.621180    2579 scope.go:117] "RemoveContainer" containerID="937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b"
Jul 28 18:21:32.623458 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.623437    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094794df-2957-4248-86bc-7d78a9bfca97-tls-certs\") pod \"094794df-2957-4248-86bc-7d78a9bfca97\" (UID: \"094794df-2957-4248-86bc-7d78a9bfca97\") "
Jul 28 18:21:32.623565 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.623471    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d78p8\" (UniqueName: \"kubernetes.io/projected/094794df-2957-4248-86bc-7d78a9bfca97-kube-api-access-d78p8\") pod \"094794df-2957-4248-86bc-7d78a9bfca97\" (UID: \"094794df-2957-4248-86bc-7d78a9bfca97\") "
Jul 28 18:21:32.625548 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.625525    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094794df-2957-4248-86bc-7d78a9bfca97-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "094794df-2957-4248-86bc-7d78a9bfca97" (UID: "094794df-2957-4248-86bc-7d78a9bfca97"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:21:32.625747 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.625724    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094794df-2957-4248-86bc-7d78a9bfca97-kube-api-access-d78p8" (OuterVolumeSpecName: "kube-api-access-d78p8") pod "094794df-2957-4248-86bc-7d78a9bfca97" (UID: "094794df-2957-4248-86bc-7d78a9bfca97"). InnerVolumeSpecName "kube-api-access-d78p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:21:32.642547 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.642528    2579 scope.go:117] "RemoveContainer" containerID="937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b"
Jul 28 18:21:32.642816 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:21:32.642796    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b\": container with ID starting with 937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b not found: ID does not exist" containerID="937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b"
Jul 28 18:21:32.642870 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.642824    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b"} err="failed to get container status \"937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b\": rpc error: code = NotFound desc = could not find container \"937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b\": container with ID starting with 937503875d65ab8f26048537eeeaa9b6befbdfe0b304bffff997b94e1b65b69b not found: ID does not exist"
Jul 28 18:21:32.725282 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.725257    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094794df-2957-4248-86bc-7d78a9bfca97-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:21:32.725282 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.725280    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d78p8\" (UniqueName: \"kubernetes.io/projected/094794df-2957-4248-86bc-7d78a9bfca97-kube-api-access-d78p8\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:21:32.943479 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.943447    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"]
Jul 28 18:21:32.947595 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:32.947567    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-59d79f7cfc-jsqdd"]
Jul 28 18:21:34.319614 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:34.319576    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="094794df-2957-4248-86bc-7d78a9bfca97" path="/var/lib/kubelet/pods/094794df-2957-4248-86bc-7d78a9bfca97/volumes"
Jul 28 18:21:37.839479 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.839441    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"]
Jul 28 18:21:37.841853 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.839843    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f815cb3-bd99-4d9c-ac10-47af5bc052ac" containerName="main"
Jul 28 18:21:37.841853 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.839855    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f815cb3-bd99-4d9c-ac10-47af5bc052ac" containerName="main"
Jul 28 18:21:37.841853 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.839864    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094794df-2957-4248-86bc-7d78a9bfca97" containerName="main"
Jul 28 18:21:37.841853 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.839873    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="094794df-2957-4248-86bc-7d78a9bfca97" containerName="main"
Jul 28 18:21:37.841853 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.839942    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f815cb3-bd99-4d9c-ac10-47af5bc052ac" containerName="main"
Jul 28 18:21:37.841853 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.839960    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="094794df-2957-4248-86bc-7d78a9bfca97" containerName="main"
Jul 28 18:21:37.843058 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.843043    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"
Jul 28 18:21:37.847074 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.847052    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-7fvqm\""
Jul 28 18:21:37.847074 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.847061    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 28 18:21:37.853726 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.853702    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"]
Jul 28 18:21:37.971471 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.971436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43911596-dc27-4427-b177-8e3d89f8cbf5-cert\") pod \"llmisvc-controller-manager-84fb8458f8-4rlxl\" (UID: \"43911596-dc27-4427-b177-8e3d89f8cbf5\") " pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"
Jul 28 18:21:37.971662 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:37.971554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtdg\" (UniqueName: \"kubernetes.io/projected/43911596-dc27-4427-b177-8e3d89f8cbf5-kube-api-access-mrtdg\") pod \"llmisvc-controller-manager-84fb8458f8-4rlxl\" (UID: \"43911596-dc27-4427-b177-8e3d89f8cbf5\") " pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"
Jul 28 18:21:38.072948 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:38.072908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtdg\" (UniqueName: \"kubernetes.io/projected/43911596-dc27-4427-b177-8e3d89f8cbf5-kube-api-access-mrtdg\") pod \"llmisvc-controller-manager-84fb8458f8-4rlxl\" (UID: \"43911596-dc27-4427-b177-8e3d89f8cbf5\") " pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"
Jul 28 18:21:38.073162 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:38.072962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43911596-dc27-4427-b177-8e3d89f8cbf5-cert\") pod \"llmisvc-controller-manager-84fb8458f8-4rlxl\" (UID: \"43911596-dc27-4427-b177-8e3d89f8cbf5\") " pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"
Jul 28 18:21:38.075469 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:38.075448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/43911596-dc27-4427-b177-8e3d89f8cbf5-cert\") pod \"llmisvc-controller-manager-84fb8458f8-4rlxl\" (UID: \"43911596-dc27-4427-b177-8e3d89f8cbf5\") " pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"
Jul 28 18:21:38.082630 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:38.082602    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtdg\" (UniqueName: \"kubernetes.io/projected/43911596-dc27-4427-b177-8e3d89f8cbf5-kube-api-access-mrtdg\") pod \"llmisvc-controller-manager-84fb8458f8-4rlxl\" (UID: \"43911596-dc27-4427-b177-8e3d89f8cbf5\") " pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"
Jul 28 18:21:38.152817 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:38.152732    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"
Jul 28 18:21:38.302650 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:38.302622    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"]
Jul 28 18:21:38.304227 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:21:38.304203    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod43911596_dc27_4427_b177_8e3d89f8cbf5.slice/crio-da9d849213abeab27b5449f7b92f6a5f7edc56151124bf6e0619afd6e0b0b994 WatchSource:0}: Error finding container da9d849213abeab27b5449f7b92f6a5f7edc56151124bf6e0619afd6e0b0b994: Status 404 returned error can't find the container with id da9d849213abeab27b5449f7b92f6a5f7edc56151124bf6e0619afd6e0b0b994
Jul 28 18:21:38.305556 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:38.305531    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:21:38.652829 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:38.652795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl" event={"ID":"43911596-dc27-4427-b177-8e3d89f8cbf5","Type":"ContainerStarted","Data":"da9d849213abeab27b5449f7b92f6a5f7edc56151124bf6e0619afd6e0b0b994"}
Jul 28 18:21:41.670527 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:41.670493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl" event={"ID":"43911596-dc27-4427-b177-8e3d89f8cbf5","Type":"ContainerStarted","Data":"dd42e1a93a371834cb9cdf8266178d540f557e09b4a8acedec132d30b07b898a"}
Jul 28 18:21:41.670965 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:41.670584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"
Jul 28 18:21:41.688107 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:21:41.688042    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl" podStartSLOduration=1.4370335650000001 podStartE2EDuration="4.688027152s" totalImagesPullingTime="3.250993587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:21:41.685865693 +0000 UTC m=+4289.915064806" watchObservedRunningTime="2026-07-28 18:21:41.688027152 +0000 UTC m=+4289.917226294"
Jul 28 18:22:12.677894 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:12.677820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-84fb8458f8-4rlxl"
Jul 28 18:22:56.515579 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.515541    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"]
Jul 28 18:22:56.519101 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.519056    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.521951 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.521924    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Jul 28 18:22:56.522236 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.522215    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Jul 28 18:22:56.523257 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.523238    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-d7dnk\""
Jul 28 18:22:56.523397 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.523334    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v1-kserve-self-signed-certs\""
Jul 28 18:22:56.529172 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.529147    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"]
Jul 28 18:22:56.670921 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.670882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-dshm\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.671120 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.670939    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.671120 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.670998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2456ac6a-946b-4c5f-933f-fb52238ffa90-tls-certs\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.671120 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.671027    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r8ng\" (UniqueName: \"kubernetes.io/projected/2456ac6a-946b-4c5f-933f-fb52238ffa90-kube-api-access-6r8ng\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.671120 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.671061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-model-cache\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.671270 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.671121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-home\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.772116 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.771986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-dshm\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.772116 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.772060    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.772375 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.772130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2456ac6a-946b-4c5f-933f-fb52238ffa90-tls-certs\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.772375 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.772153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r8ng\" (UniqueName: \"kubernetes.io/projected/2456ac6a-946b-4c5f-933f-fb52238ffa90-kube-api-access-6r8ng\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.772375 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.772183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-model-cache\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.772375 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.772238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-home\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.772592 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.772510    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-tmp-dir\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.772592 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.772562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-model-cache\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.772700 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.772653    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-home\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.774296 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.774276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-dshm\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.774695 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.774676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2456ac6a-946b-4c5f-933f-fb52238ffa90-tls-certs\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.780694 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.780667    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r8ng\" (UniqueName: \"kubernetes.io/projected/2456ac6a-946b-4c5f-933f-fb52238ffa90-kube-api-access-6r8ng\") pod \"leave-v1-kserve-65684d4f5-vvwfp\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") " pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.832409 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.832382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:22:56.968486 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:56.968460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"]
Jul 28 18:22:56.970620 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:22:56.970591    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2456ac6a_946b_4c5f_933f_fb52238ffa90.slice/crio-d6180f98f3faf873bc43f84875db6b510130816622ac17a737887f15230f065d WatchSource:0}: Error finding container d6180f98f3faf873bc43f84875db6b510130816622ac17a737887f15230f065d: Status 404 returned error can't find the container with id d6180f98f3faf873bc43f84875db6b510130816622ac17a737887f15230f065d
Jul 28 18:22:57.037185 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:57.037097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp" event={"ID":"2456ac6a-946b-4c5f-933f-fb52238ffa90","Type":"ContainerStarted","Data":"d6180f98f3faf873bc43f84875db6b510130816622ac17a737887f15230f065d"}
Jul 28 18:22:59.048801 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:59.048760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp" event={"ID":"2456ac6a-946b-4c5f-933f-fb52238ffa90","Type":"ContainerStarted","Data":"4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404"}
Jul 28 18:22:59.072122 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:59.072030    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp" podStartSLOduration=2.051072444 podStartE2EDuration="3.072016675s" totalImagesPullingTime="1.020944231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:22:59.070607804 +0000 UTC m=+4367.299806920" watchObservedRunningTime="2026-07-28 18:22:59.072016675 +0000 UTC m=+4367.301215770"
Jul 28 18:22:59.833526 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:22:59.833488    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:23:06.832851 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:06.832815    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:23:06.845097 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:06.845056    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:23:07.100446 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.100372    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:23:07.709134 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.709101    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"]
Jul 28 18:23:07.711919 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.711900    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.714818 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.714791    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Jul 28 18:23:07.722446 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.722424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"]
Jul 28 18:23:07.876054 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.876020    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-model-cache\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.876470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.876105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-home\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.876470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.876139    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5m9\" (UniqueName: \"kubernetes.io/projected/18e1a91c-eaca-49a8-b865-83acef15e216-kube-api-access-gf5m9\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.876470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.876166    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-dshm\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.876470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.876263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e1a91c-eaca-49a8-b865-83acef15e216-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.876470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.876281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.976964 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.976885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e1a91c-eaca-49a8-b865-83acef15e216-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.976964 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.976920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.976964 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.976941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-model-cache\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.977242 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.976982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-home\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.977242 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.977004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5m9\" (UniqueName: \"kubernetes.io/projected/18e1a91c-eaca-49a8-b865-83acef15e216-kube-api-access-gf5m9\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.977242 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.977030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-dshm\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.977407 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.977382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.977470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.977414    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-model-cache\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.977470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.977451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-home\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.979364 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.979344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-dshm\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.979635 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.979617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e1a91c-eaca-49a8-b865-83acef15e216-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:07.985693 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:07.985666    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5m9\" (UniqueName: \"kubernetes.io/projected/18e1a91c-eaca-49a8-b865-83acef15e216-kube-api-access-gf5m9\") pod \"leave-v2-kserve-556b4dddf9-mxz7k\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:08.023317 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:08.023275    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:08.161383 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:08.161359    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"]
Jul 28 18:23:08.163500 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:23:08.163466    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e1a91c_eaca_49a8_b865_83acef15e216.slice/crio-2d68ec8403e3ec608ff9157881833256399ed8b509af897dfe2708fbacbeb7e4 WatchSource:0}: Error finding container 2d68ec8403e3ec608ff9157881833256399ed8b509af897dfe2708fbacbeb7e4: Status 404 returned error can't find the container with id 2d68ec8403e3ec608ff9157881833256399ed8b509af897dfe2708fbacbeb7e4
Jul 28 18:23:09.104154 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:09.104114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k" event={"ID":"18e1a91c-eaca-49a8-b865-83acef15e216","Type":"ContainerStarted","Data":"de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39"}
Jul 28 18:23:09.104565 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:09.104176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k" event={"ID":"18e1a91c-eaca-49a8-b865-83acef15e216","Type":"ContainerStarted","Data":"2d68ec8403e3ec608ff9157881833256399ed8b509af897dfe2708fbacbeb7e4"}
Jul 28 18:23:09.124762 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:09.124694    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k" podStartSLOduration=2.124674085 podStartE2EDuration="2.124674085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:23:09.124542397 +0000 UTC m=+4377.353741495" watchObservedRunningTime="2026-07-28 18:23:09.124674085 +0000 UTC m=+4377.353873184"
Jul 28 18:23:10.024055 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:10.024020    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:18.023771 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:18.023732    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:18.036369 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:18.036344    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:18.161957 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:18.161925    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:35.621802 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:35.621722    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"]
Jul 28 18:23:35.626644 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:35.626615    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"]
Jul 28 18:23:44.508196 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.508162    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"]
Jul 28 18:23:44.510981 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.510961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.514739 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.514715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"default-dockercfg-bwqbj\""
Jul 28 18:23:44.514739 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.514735    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"openshift-service-ca.crt\""
Jul 28 18:23:44.514927 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.514715    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v2-kserve-self-signed-certs\""
Jul 28 18:23:44.514927 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.514718    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"kube-root-ca.crt\""
Jul 28 18:23:44.523027 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.523006    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"]
Jul 28 18:23:44.603751 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.603715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6bs2\" (UniqueName: \"kubernetes.io/projected/c4643423-d47e-41eb-81fa-130889d4e6e0-kube-api-access-w6bs2\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.603903 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.603773    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-dshm\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.603903 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.603811    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-home\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.603903 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.603850    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.603903 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.603879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4643423-d47e-41eb-81fa-130889d4e6e0-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.604068 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.603903    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-model-cache\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.704770 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.704732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6bs2\" (UniqueName: \"kubernetes.io/projected/c4643423-d47e-41eb-81fa-130889d4e6e0-kube-api-access-w6bs2\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.704951 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.704795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-dshm\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.704951 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.704823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-home\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.704951 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.704864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.705152 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.704992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4643423-d47e-41eb-81fa-130889d4e6e0-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.705152 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.705028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-model-cache\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.705274 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.705261    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-home\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.705348 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.705326    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-tmp-dir\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.705398 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.705366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-model-cache\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.707008 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.706981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-dshm\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.707416 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.707398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4643423-d47e-41eb-81fa-130889d4e6e0-tls-certs\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.713194 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.713177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6bs2\" (UniqueName: \"kubernetes.io/projected/c4643423-d47e-41eb-81fa-130889d4e6e0-kube-api-access-w6bs2\") pod \"tri-v2-kserve-dcb554bf5-76rtb\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") " pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.823696 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.823665    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:44.961050 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:44.961019    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"]
Jul 28 18:23:44.963804 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:23:44.963775    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4643423_d47e_41eb_81fa_130889d4e6e0.slice/crio-8ce64ca6c6daf3e31c55cc86706ecfb67cf22231f75133042180db07675588eb WatchSource:0}: Error finding container 8ce64ca6c6daf3e31c55cc86706ecfb67cf22231f75133042180db07675588eb: Status 404 returned error can't find the container with id 8ce64ca6c6daf3e31c55cc86706ecfb67cf22231f75133042180db07675588eb
Jul 28 18:23:45.288819 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:45.288735    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb" event={"ID":"c4643423-d47e-41eb-81fa-130889d4e6e0","Type":"ContainerStarted","Data":"9e1b9744809a9798233f88f7033967d6964ac71278a873d77442f7ce166a3203"}
Jul 28 18:23:45.288819 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:45.288790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb" event={"ID":"c4643423-d47e-41eb-81fa-130889d4e6e0","Type":"ContainerStarted","Data":"8ce64ca6c6daf3e31c55cc86706ecfb67cf22231f75133042180db07675588eb"}
Jul 28 18:23:45.308446 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:45.308403    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb" podStartSLOduration=1.308388166 podStartE2EDuration="1.308388166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:23:45.306421009 +0000 UTC m=+4413.535620099" watchObservedRunningTime="2026-07-28 18:23:45.308388166 +0000 UTC m=+4413.537587261"
Jul 28 18:23:45.824163 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:45.824133    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:50.641618 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:50.641561    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp" podUID="2456ac6a-946b-4c5f-933f-fb52238ffa90" containerName="main" containerID="cri-o://4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404" gracePeriod=45
Jul 28 18:23:50.645682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:50.645647    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k" podUID="18e1a91c-eaca-49a8-b865-83acef15e216" containerName="main" containerID="cri-o://de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39" gracePeriod=45
Jul 28 18:23:50.920130 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:50.920104    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:23:50.922969 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:50.922947    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:51.066502 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066465    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2456ac6a-946b-4c5f-933f-fb52238ffa90-tls-certs\") pod \"2456ac6a-946b-4c5f-933f-fb52238ffa90\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") "
Jul 28 18:23:51.066704 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066592    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf5m9\" (UniqueName: \"kubernetes.io/projected/18e1a91c-eaca-49a8-b865-83acef15e216-kube-api-access-gf5m9\") pod \"18e1a91c-eaca-49a8-b865-83acef15e216\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") "
Jul 28 18:23:51.066704 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e1a91c-eaca-49a8-b865-83acef15e216-tls-certs\") pod \"18e1a91c-eaca-49a8-b865-83acef15e216\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") "
Jul 28 18:23:51.066704 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066680    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-tmp-dir\") pod \"2456ac6a-946b-4c5f-933f-fb52238ffa90\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") "
Jul 28 18:23:51.066894 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066714    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-model-cache\") pod \"18e1a91c-eaca-49a8-b865-83acef15e216\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") "
Jul 28 18:23:51.066894 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066740    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-home\") pod \"2456ac6a-946b-4c5f-933f-fb52238ffa90\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") "
Jul 28 18:23:51.066894 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066789    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-home\") pod \"18e1a91c-eaca-49a8-b865-83acef15e216\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") "
Jul 28 18:23:51.066894 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r8ng\" (UniqueName: \"kubernetes.io/projected/2456ac6a-946b-4c5f-933f-fb52238ffa90-kube-api-access-6r8ng\") pod \"2456ac6a-946b-4c5f-933f-fb52238ffa90\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") "
Jul 28 18:23:51.066894 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066848    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-tmp-dir\") pod \"18e1a91c-eaca-49a8-b865-83acef15e216\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") "
Jul 28 18:23:51.066894 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066881    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-dshm\") pod \"18e1a91c-eaca-49a8-b865-83acef15e216\" (UID: \"18e1a91c-eaca-49a8-b865-83acef15e216\") "
Jul 28 18:23:51.067216 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066913    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-dshm\") pod \"2456ac6a-946b-4c5f-933f-fb52238ffa90\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") "
Jul 28 18:23:51.067216 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.066950    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-model-cache\") pod \"2456ac6a-946b-4c5f-933f-fb52238ffa90\" (UID: \"2456ac6a-946b-4c5f-933f-fb52238ffa90\") "
Jul 28 18:23:51.067216 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.067009    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2456ac6a-946b-4c5f-933f-fb52238ffa90" (UID: "2456ac6a-946b-4c5f-933f-fb52238ffa90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:23:51.067216 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.067009    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-model-cache" (OuterVolumeSpecName: "model-cache") pod "18e1a91c-eaca-49a8-b865-83acef15e216" (UID: "18e1a91c-eaca-49a8-b865-83acef15e216"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:23:51.067446 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.067228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-home" (OuterVolumeSpecName: "home") pod "2456ac6a-946b-4c5f-933f-fb52238ffa90" (UID: "2456ac6a-946b-4c5f-933f-fb52238ffa90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:23:51.067446 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.067349    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.067446 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.067372    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.067446 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.067385    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.067797 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.067448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-model-cache" (OuterVolumeSpecName: "model-cache") pod "2456ac6a-946b-4c5f-933f-fb52238ffa90" (UID: "2456ac6a-946b-4c5f-933f-fb52238ffa90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:23:51.067797 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.067453    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18e1a91c-eaca-49a8-b865-83acef15e216" (UID: "18e1a91c-eaca-49a8-b865-83acef15e216"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:23:51.067797 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.067638    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-home" (OuterVolumeSpecName: "home") pod "18e1a91c-eaca-49a8-b865-83acef15e216" (UID: "18e1a91c-eaca-49a8-b865-83acef15e216"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:23:51.069247 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.069215    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e1a91c-eaca-49a8-b865-83acef15e216-kube-api-access-gf5m9" (OuterVolumeSpecName: "kube-api-access-gf5m9") pod "18e1a91c-eaca-49a8-b865-83acef15e216" (UID: "18e1a91c-eaca-49a8-b865-83acef15e216"). InnerVolumeSpecName "kube-api-access-gf5m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:23:51.069247 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.069228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2456ac6a-946b-4c5f-933f-fb52238ffa90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2456ac6a-946b-4c5f-933f-fb52238ffa90" (UID: "2456ac6a-946b-4c5f-933f-fb52238ffa90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:23:51.069415 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.069262    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e1a91c-eaca-49a8-b865-83acef15e216-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18e1a91c-eaca-49a8-b865-83acef15e216" (UID: "18e1a91c-eaca-49a8-b865-83acef15e216"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:23:51.069474 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.069457    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-dshm" (OuterVolumeSpecName: "dshm") pod "2456ac6a-946b-4c5f-933f-fb52238ffa90" (UID: "2456ac6a-946b-4c5f-933f-fb52238ffa90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:23:51.069524 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.069506    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-dshm" (OuterVolumeSpecName: "dshm") pod "18e1a91c-eaca-49a8-b865-83acef15e216" (UID: "18e1a91c-eaca-49a8-b865-83acef15e216"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:23:51.069609 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.069592    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2456ac6a-946b-4c5f-933f-fb52238ffa90-kube-api-access-6r8ng" (OuterVolumeSpecName: "kube-api-access-6r8ng") pod "2456ac6a-946b-4c5f-933f-fb52238ffa90" (UID: "2456ac6a-946b-4c5f-933f-fb52238ffa90"). InnerVolumeSpecName "kube-api-access-6r8ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:23:51.169043 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.168971    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.169043 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.168996    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r8ng\" (UniqueName: \"kubernetes.io/projected/2456ac6a-946b-4c5f-933f-fb52238ffa90-kube-api-access-6r8ng\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.169043 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.169009    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.169043 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.169019    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18e1a91c-eaca-49a8-b865-83acef15e216-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.169043 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.169027    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.169043 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.169035    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2456ac6a-946b-4c5f-933f-fb52238ffa90-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.169043 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.169043    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2456ac6a-946b-4c5f-933f-fb52238ffa90-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.169043 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.169050    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf5m9\" (UniqueName: \"kubernetes.io/projected/18e1a91c-eaca-49a8-b865-83acef15e216-kube-api-access-gf5m9\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.169428 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.169058    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18e1a91c-eaca-49a8-b865-83acef15e216-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:23:51.319022 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.318987    2579 generic.go:356] "Generic (PLEG): container finished" podID="2456ac6a-946b-4c5f-933f-fb52238ffa90" containerID="4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404" exitCode=0
Jul 28 18:23:51.319221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.319036    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"
Jul 28 18:23:51.319221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.319064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp" event={"ID":"2456ac6a-946b-4c5f-933f-fb52238ffa90","Type":"ContainerDied","Data":"4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404"}
Jul 28 18:23:51.319221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.319131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp" event={"ID":"2456ac6a-946b-4c5f-933f-fb52238ffa90","Type":"ContainerDied","Data":"d6180f98f3faf873bc43f84875db6b510130816622ac17a737887f15230f065d"}
Jul 28 18:23:51.319221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.319164    2579 scope.go:117] "RemoveContainer" containerID="4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404"
Jul 28 18:23:51.320399 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.320375    2579 generic.go:356] "Generic (PLEG): container finished" podID="18e1a91c-eaca-49a8-b865-83acef15e216" containerID="de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39" exitCode=0
Jul 28 18:23:51.320476 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.320410    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"
Jul 28 18:23:51.320476 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.320417    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k" event={"ID":"18e1a91c-eaca-49a8-b865-83acef15e216","Type":"ContainerDied","Data":"de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39"}
Jul 28 18:23:51.320476 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.320455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k" event={"ID":"18e1a91c-eaca-49a8-b865-83acef15e216","Type":"ContainerDied","Data":"2d68ec8403e3ec608ff9157881833256399ed8b509af897dfe2708fbacbeb7e4"}
Jul 28 18:23:51.328429 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.328359    2579 scope.go:117] "RemoveContainer" containerID="4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404"
Jul 28 18:23:51.328666 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:23:51.328644    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404\": container with ID starting with 4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404 not found: ID does not exist" containerID="4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404"
Jul 28 18:23:51.328751 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.328676    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404"} err="failed to get container status \"4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404\": rpc error: code = NotFound desc = could not find container \"4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404\": container with ID starting with 4b019ffdfd42ef565ca4c640608eef496c9ecb0687d3fb914d86798d4742d404 not found: ID does not exist"
Jul 28 18:23:51.328751 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.328698    2579 scope.go:117] "RemoveContainer" containerID="de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39"
Jul 28 18:23:51.337015 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.336980    2579 scope.go:117] "RemoveContainer" containerID="de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39"
Jul 28 18:23:51.337269 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:23:51.337246    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39\": container with ID starting with de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39 not found: ID does not exist" containerID="de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39"
Jul 28 18:23:51.337348 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.337275    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39"} err="failed to get container status \"de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39\": rpc error: code = NotFound desc = could not find container \"de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39\": container with ID starting with de7e47e24afe654957b1240270c07e29435baf5b8f010d86e86780ce719f3d39 not found: ID does not exist"
Jul 28 18:23:51.346783 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.346755    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"]
Jul 28 18:23:51.349147 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.349129    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v1-kserve-65684d4f5-vvwfp"]
Jul 28 18:23:51.360699 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.360677    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"]
Jul 28 18:23:51.365331 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:51.365307    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-mxz7k"]
Jul 28 18:23:52.320202 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:52.320172    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e1a91c-eaca-49a8-b865-83acef15e216" path="/var/lib/kubelet/pods/18e1a91c-eaca-49a8-b865-83acef15e216/volumes"
Jul 28 18:23:52.320610 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:52.320594    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2456ac6a-946b-4c5f-933f-fb52238ffa90" path="/var/lib/kubelet/pods/2456ac6a-946b-4c5f-933f-fb52238ffa90/volumes"
Jul 28 18:23:54.824505 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:54.824473    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:54.836888 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:54.836867    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:23:55.353826 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:23:55.353797    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:24:22.890826 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.890789    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"]
Jul 28 18:24:22.891334 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.891235    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e1a91c-eaca-49a8-b865-83acef15e216" containerName="main"
Jul 28 18:24:22.891334 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.891249    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e1a91c-eaca-49a8-b865-83acef15e216" containerName="main"
Jul 28 18:24:22.891334 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.891264    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2456ac6a-946b-4c5f-933f-fb52238ffa90" containerName="main"
Jul 28 18:24:22.891334 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.891269    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2456ac6a-946b-4c5f-933f-fb52238ffa90" containerName="main"
Jul 28 18:24:22.891483 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.891355    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2456ac6a-946b-4c5f-933f-fb52238ffa90" containerName="main"
Jul 28 18:24:22.891483 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.891365    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e1a91c-eaca-49a8-b865-83acef15e216" containerName="main"
Jul 28 18:24:22.894836 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.894817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:22.897574 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.897555    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-three-member-group-c33d828b\"/\"tri-v3-kserve-self-signed-certs\""
Jul 28 18:24:22.905239 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.905217    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"]
Jul 28 18:24:22.956767 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.956730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgst8\" (UniqueName: \"kubernetes.io/projected/21aeb183-f8c0-4271-a01c-f633c4eb7637-kube-api-access-mgst8\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:22.956767 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.956772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-dshm\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:22.956962 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.956792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:22.956962 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.956815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-home\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:22.956962 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.956861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21aeb183-f8c0-4271-a01c-f633c4eb7637-tls-certs\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:22.956962 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:22.956917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-model-cache\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.057538 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.057494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-model-cache\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.057714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.057596    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgst8\" (UniqueName: \"kubernetes.io/projected/21aeb183-f8c0-4271-a01c-f633c4eb7637-kube-api-access-mgst8\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.057714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.057623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-dshm\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.057714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.057640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.057714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.057666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-home\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.057714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.057690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21aeb183-f8c0-4271-a01c-f633c4eb7637-tls-certs\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.057990 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.057963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-model-cache\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.058054 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.057975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-tmp-dir\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.058187 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.058166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-home\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.059944 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.059922    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-dshm\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.060218 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.060202    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21aeb183-f8c0-4271-a01c-f633c4eb7637-tls-certs\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.065265 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.065247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgst8\" (UniqueName: \"kubernetes.io/projected/21aeb183-f8c0-4271-a01c-f633c4eb7637-kube-api-access-mgst8\") pod \"tri-v3-kserve-798c88c66c-h92zb\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") " pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.204973 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.204887    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:23.337685 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.337660    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"]
Jul 28 18:24:23.340046 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:24:23.340012    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21aeb183_f8c0_4271_a01c_f633c4eb7637.slice/crio-cd9524c5226ee05eb1d680527538da0963520baf4aa0bd1a5fde8a940f783a9e WatchSource:0}: Error finding container cd9524c5226ee05eb1d680527538da0963520baf4aa0bd1a5fde8a940f783a9e: Status 404 returned error can't find the container with id cd9524c5226ee05eb1d680527538da0963520baf4aa0bd1a5fde8a940f783a9e
Jul 28 18:24:23.482414 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.482338    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb" event={"ID":"21aeb183-f8c0-4271-a01c-f633c4eb7637","Type":"ContainerStarted","Data":"57ab77cec1cf17cf96c4701d7955c35809c76b23bde5ca6b6005271096c3f30e"}
Jul 28 18:24:23.482414 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.482399    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb" event={"ID":"21aeb183-f8c0-4271-a01c-f633c4eb7637","Type":"ContainerStarted","Data":"cd9524c5226ee05eb1d680527538da0963520baf4aa0bd1a5fde8a940f783a9e"}
Jul 28 18:24:23.500938 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:23.500893    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb" podStartSLOduration=1.5008788389999999 podStartE2EDuration="1.500878839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:24:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:24:23.500226731 +0000 UTC m=+4451.729425827" watchObservedRunningTime="2026-07-28 18:24:23.500878839 +0000 UTC m=+4451.730077935"
Jul 28 18:24:24.205394 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:24.205354    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:33.205979 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:33.205938    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:33.218393 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:33.218365    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:24:33.541902 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:24:33.541828    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:25:12.623455 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:12.623391    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:25:12.649362 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:12.649337    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:25:51.759929 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:51.759893    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"]
Jul 28 18:25:51.771614 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:51.771588    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"]
Jul 28 18:25:55.014139 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.014106    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"]
Jul 28 18:25:55.016963 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.016944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.020657 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.020632    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Jul 28 18:25:55.020769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.020662    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-bqf79\""
Jul 28 18:25:55.020769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.020648    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Jul 28 18:25:55.020769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.020694    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Jul 28 18:25:55.030698 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.030674    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"]
Jul 28 18:25:55.091661 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.091628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-dshm\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.091809 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.091667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-model-cache\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.091809 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.091696    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.091809 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.091786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vq4d\" (UniqueName: \"kubernetes.io/projected/736e9978-796f-4d00-8366-bde5f348e2ad-kube-api-access-5vq4d\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.091969 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.091830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-home\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.091969 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.091908    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736e9978-796f-4d00-8366-bde5f348e2ad-tls-certs\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.192855 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.192822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-home\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.192992 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.192864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736e9978-796f-4d00-8366-bde5f348e2ad-tls-certs\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.192992 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.192910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-dshm\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.192992 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.192937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-model-cache\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.192992 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.192990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.193175 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.193064    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vq4d\" (UniqueName: \"kubernetes.io/projected/736e9978-796f-4d00-8366-bde5f348e2ad-kube-api-access-5vq4d\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.193328 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.193308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-home\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.193373 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.193351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-model-cache\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.193433 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.193407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.195306 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.195284    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-dshm\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.195502 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.195485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736e9978-796f-4d00-8366-bde5f348e2ad-tls-certs\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.201481 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.201458    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vq4d\" (UniqueName: \"kubernetes.io/projected/736e9978-796f-4d00-8366-bde5f348e2ad-kube-api-access-5vq4d\") pod \"late-v1-kserve-56cbc8456c-zqssn\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.327702 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.327671    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:25:55.462588 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.462562    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"]
Jul 28 18:25:55.464723 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:25:55.464695    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod736e9978_796f_4d00_8366_bde5f348e2ad.slice/crio-bac21b0d41af50b34de43f875f202a1c1f37d4651ecdbdcd925bf5b6a318a27b WatchSource:0}: Error finding container bac21b0d41af50b34de43f875f202a1c1f37d4651ecdbdcd925bf5b6a318a27b: Status 404 returned error can't find the container with id bac21b0d41af50b34de43f875f202a1c1f37d4651ecdbdcd925bf5b6a318a27b
Jul 28 18:25:55.943847 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.943811    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn" event={"ID":"736e9978-796f-4d00-8366-bde5f348e2ad","Type":"ContainerStarted","Data":"94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196"}
Jul 28 18:25:55.944226 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.943875    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn" event={"ID":"736e9978-796f-4d00-8366-bde5f348e2ad","Type":"ContainerStarted","Data":"bac21b0d41af50b34de43f875f202a1c1f37d4651ecdbdcd925bf5b6a318a27b"}
Jul 28 18:25:55.962346 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:55.962299    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn" podStartSLOduration=1.962282667 podStartE2EDuration="1.962282667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:25:55.961566509 +0000 UTC m=+4544.190765747" watchObservedRunningTime="2026-07-28 18:25:55.962282667 +0000 UTC m=+4544.191481762"
Jul 28 18:25:56.328508 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:25:56.328482    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:26:05.328578 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:05.328543    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:26:05.340942 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:05.340914    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:26:06.004561 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:06.004536    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:26:06.781111 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:06.781048    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb" podUID="c4643423-d47e-41eb-81fa-130889d4e6e0" containerName="main" containerID="cri-o://9e1b9744809a9798233f88f7033967d6964ac71278a873d77442f7ce166a3203" gracePeriod=45
Jul 28 18:26:06.791699 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:06.791657    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb" podUID="21aeb183-f8c0-4271-a01c-f633c4eb7637" containerName="main" containerID="cri-o://57ab77cec1cf17cf96c4701d7955c35809c76b23bde5ca6b6005271096c3f30e" gracePeriod=45
Jul 28 18:26:07.003715 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.003598    2579 generic.go:356] "Generic (PLEG): container finished" podID="21aeb183-f8c0-4271-a01c-f633c4eb7637" containerID="57ab77cec1cf17cf96c4701d7955c35809c76b23bde5ca6b6005271096c3f30e" exitCode=0
Jul 28 18:26:07.003715 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.003690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb" event={"ID":"21aeb183-f8c0-4271-a01c-f633c4eb7637","Type":"ContainerDied","Data":"57ab77cec1cf17cf96c4701d7955c35809c76b23bde5ca6b6005271096c3f30e"}
Jul 28 18:26:07.013033 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.013005    2579 generic.go:356] "Generic (PLEG): container finished" podID="c4643423-d47e-41eb-81fa-130889d4e6e0" containerID="9e1b9744809a9798233f88f7033967d6964ac71278a873d77442f7ce166a3203" exitCode=0
Jul 28 18:26:07.013195 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.013068    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb" event={"ID":"c4643423-d47e-41eb-81fa-130889d4e6e0","Type":"ContainerDied","Data":"9e1b9744809a9798233f88f7033967d6964ac71278a873d77442f7ce166a3203"}
Jul 28 18:26:07.067246 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.067221    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:26:07.070475 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.070453    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:26:07.209809 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.209769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-model-cache\") pod \"21aeb183-f8c0-4271-a01c-f633c4eb7637\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") "
Jul 28 18:26:07.209809 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.209818    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgst8\" (UniqueName: \"kubernetes.io/projected/21aeb183-f8c0-4271-a01c-f633c4eb7637-kube-api-access-mgst8\") pod \"21aeb183-f8c0-4271-a01c-f633c4eb7637\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.209837    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-home\") pod \"c4643423-d47e-41eb-81fa-130889d4e6e0\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.209860    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-tmp-dir\") pod \"c4643423-d47e-41eb-81fa-130889d4e6e0\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.209902    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6bs2\" (UniqueName: \"kubernetes.io/projected/c4643423-d47e-41eb-81fa-130889d4e6e0-kube-api-access-w6bs2\") pod \"c4643423-d47e-41eb-81fa-130889d4e6e0\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.209918    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-home\") pod \"21aeb183-f8c0-4271-a01c-f633c4eb7637\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.209936    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21aeb183-f8c0-4271-a01c-f633c4eb7637-tls-certs\") pod \"21aeb183-f8c0-4271-a01c-f633c4eb7637\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.209997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-tmp-dir\") pod \"21aeb183-f8c0-4271-a01c-f633c4eb7637\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.210027    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-dshm\") pod \"c4643423-d47e-41eb-81fa-130889d4e6e0\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.210045    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-home" (OuterVolumeSpecName: "home") pod "c4643423-d47e-41eb-81fa-130889d4e6e0" (UID: "c4643423-d47e-41eb-81fa-130889d4e6e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.210057    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-model-cache" (OuterVolumeSpecName: "model-cache") pod "21aeb183-f8c0-4271-a01c-f633c4eb7637" (UID: "21aeb183-f8c0-4271-a01c-f633c4eb7637"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.210105    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-model-cache\") pod \"c4643423-d47e-41eb-81fa-130889d4e6e0\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.210129    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-dshm\") pod \"21aeb183-f8c0-4271-a01c-f633c4eb7637\" (UID: \"21aeb183-f8c0-4271-a01c-f633c4eb7637\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.210155    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4643423-d47e-41eb-81fa-130889d4e6e0-tls-certs\") pod \"c4643423-d47e-41eb-81fa-130889d4e6e0\" (UID: \"c4643423-d47e-41eb-81fa-130889d4e6e0\") "
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.210170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4643423-d47e-41eb-81fa-130889d4e6e0" (UID: "c4643423-d47e-41eb-81fa-130889d4e6e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.210235    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-home" (OuterVolumeSpecName: "home") pod "21aeb183-f8c0-4271-a01c-f633c4eb7637" (UID: "21aeb183-f8c0-4271-a01c-f633c4eb7637"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.210239    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "21aeb183-f8c0-4271-a01c-f633c4eb7637" (UID: "21aeb183-f8c0-4271-a01c-f633c4eb7637"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:07.210402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.210349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4643423-d47e-41eb-81fa-130889d4e6e0" (UID: "c4643423-d47e-41eb-81fa-130889d4e6e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:07.211377 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.211355    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.211550 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.211528    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.211550 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.211550    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.211687 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.211563    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.211687 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.211575    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.211687 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.211587    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.212570 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.212548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-dshm" (OuterVolumeSpecName: "dshm") pod "c4643423-d47e-41eb-81fa-130889d4e6e0" (UID: "c4643423-d47e-41eb-81fa-130889d4e6e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:07.212687 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.212604    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-dshm" (OuterVolumeSpecName: "dshm") pod "21aeb183-f8c0-4271-a01c-f633c4eb7637" (UID: "21aeb183-f8c0-4271-a01c-f633c4eb7637"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:26:07.212740 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.212674    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21aeb183-f8c0-4271-a01c-f633c4eb7637-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "21aeb183-f8c0-4271-a01c-f633c4eb7637" (UID: "21aeb183-f8c0-4271-a01c-f633c4eb7637"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:26:07.212740 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.212708    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21aeb183-f8c0-4271-a01c-f633c4eb7637-kube-api-access-mgst8" (OuterVolumeSpecName: "kube-api-access-mgst8") pod "21aeb183-f8c0-4271-a01c-f633c4eb7637" (UID: "21aeb183-f8c0-4271-a01c-f633c4eb7637"). InnerVolumeSpecName "kube-api-access-mgst8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:26:07.212740 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.212730    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4643423-d47e-41eb-81fa-130889d4e6e0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4643423-d47e-41eb-81fa-130889d4e6e0" (UID: "c4643423-d47e-41eb-81fa-130889d4e6e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:26:07.212848 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.212785    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4643423-d47e-41eb-81fa-130889d4e6e0-kube-api-access-w6bs2" (OuterVolumeSpecName: "kube-api-access-w6bs2") pod "c4643423-d47e-41eb-81fa-130889d4e6e0" (UID: "c4643423-d47e-41eb-81fa-130889d4e6e0"). InnerVolumeSpecName "kube-api-access-w6bs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:26:07.312838 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.312760    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgst8\" (UniqueName: \"kubernetes.io/projected/21aeb183-f8c0-4271-a01c-f633c4eb7637-kube-api-access-mgst8\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.312838 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.312787    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6bs2\" (UniqueName: \"kubernetes.io/projected/c4643423-d47e-41eb-81fa-130889d4e6e0-kube-api-access-w6bs2\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.312838 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.312797    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21aeb183-f8c0-4271-a01c-f633c4eb7637-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.312838 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.312806    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4643423-d47e-41eb-81fa-130889d4e6e0-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.312838 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.312813    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21aeb183-f8c0-4271-a01c-f633c4eb7637-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:07.312838 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:07.312820    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4643423-d47e-41eb-81fa-130889d4e6e0-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:26:08.019154 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.019123    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"
Jul 28 18:26:08.019154 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.019138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb" event={"ID":"21aeb183-f8c0-4271-a01c-f633c4eb7637","Type":"ContainerDied","Data":"cd9524c5226ee05eb1d680527538da0963520baf4aa0bd1a5fde8a940f783a9e"}
Jul 28 18:26:08.019715 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.019198    2579 scope.go:117] "RemoveContainer" containerID="57ab77cec1cf17cf96c4701d7955c35809c76b23bde5ca6b6005271096c3f30e"
Jul 28 18:26:08.020549 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.020451    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"
Jul 28 18:26:08.020549 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.020456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb" event={"ID":"c4643423-d47e-41eb-81fa-130889d4e6e0","Type":"ContainerDied","Data":"8ce64ca6c6daf3e31c55cc86706ecfb67cf22231f75133042180db07675588eb"}
Jul 28 18:26:08.029686 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.029667    2579 scope.go:117] "RemoveContainer" containerID="9e1b9744809a9798233f88f7033967d6964ac71278a873d77442f7ce166a3203"
Jul 28 18:26:08.046010 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.045980    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"]
Jul 28 18:26:08.049748 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.049721    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v2-kserve-dcb554bf5-76rtb"]
Jul 28 18:26:08.060420 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.060393    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"]
Jul 28 18:26:08.064432 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.064408    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-three-member-group-c33d828b/tri-v3-kserve-798c88c66c-h92zb"]
Jul 28 18:26:08.320189 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.320154    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21aeb183-f8c0-4271-a01c-f633c4eb7637" path="/var/lib/kubelet/pods/21aeb183-f8c0-4271-a01c-f633c4eb7637/volumes"
Jul 28 18:26:08.320594 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:08.320578    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4643423-d47e-41eb-81fa-130889d4e6e0" path="/var/lib/kubelet/pods/c4643423-d47e-41eb-81fa-130889d4e6e0/volumes"
Jul 28 18:26:32.515786 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.515747    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"]
Jul 28 18:26:32.516320 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.516148    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4643423-d47e-41eb-81fa-130889d4e6e0" containerName="main"
Jul 28 18:26:32.516320 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.516162    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4643423-d47e-41eb-81fa-130889d4e6e0" containerName="main"
Jul 28 18:26:32.516320 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.516183    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21aeb183-f8c0-4271-a01c-f633c4eb7637" containerName="main"
Jul 28 18:26:32.516320 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.516188    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="21aeb183-f8c0-4271-a01c-f633c4eb7637" containerName="main"
Jul 28 18:26:32.516320 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.516256    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="21aeb183-f8c0-4271-a01c-f633c4eb7637" containerName="main"
Jul 28 18:26:32.516320 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.516263    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4643423-d47e-41eb-81fa-130889d4e6e0" containerName="main"
Jul 28 18:26:32.519971 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.519950    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.526903 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.526643    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v2-kserve-self-signed-certs\""
Jul 28 18:26:32.535426 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.535403    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"]
Jul 28 18:26:32.646138 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.646102    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tmp-dir\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.646290 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.646156    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-dshm\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.646290 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.646175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-home\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.646290 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.646190    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-model-cache\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.646290 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.646228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tls-certs\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.646461 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.646292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xrvq\" (UniqueName: \"kubernetes.io/projected/8ba0d2a1-5796-4a6a-8367-f72f1b197613-kube-api-access-2xrvq\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.747488 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.747458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tmp-dir\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.747665 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.747505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-dshm\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.747665 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.747533    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-home\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.747794 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.747669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-model-cache\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.747794 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.747719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tls-certs\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.747794 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.747756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xrvq\" (UniqueName: \"kubernetes.io/projected/8ba0d2a1-5796-4a6a-8367-f72f1b197613-kube-api-access-2xrvq\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.747957 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.747888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-home\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.748030 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.748001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tmp-dir\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.748136 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.748120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-model-cache\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.749887 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.749859    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-dshm\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.750258 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.750234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tls-certs\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.756630 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.756609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xrvq\" (UniqueName: \"kubernetes.io/projected/8ba0d2a1-5796-4a6a-8367-f72f1b197613-kube-api-access-2xrvq\") pod \"late-v2-kserve-86766546f-xznr2\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") " pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.830740 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.830714    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:32.961415 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:32.961384    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"]
Jul 28 18:26:32.964588 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:26:32.964559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba0d2a1_5796_4a6a_8367_f72f1b197613.slice/crio-098f07cea369cd158f442c70542a1d4baa6fed7dbd798899e94d8af979af15ec WatchSource:0}: Error finding container 098f07cea369cd158f442c70542a1d4baa6fed7dbd798899e94d8af979af15ec: Status 404 returned error can't find the container with id 098f07cea369cd158f442c70542a1d4baa6fed7dbd798899e94d8af979af15ec
Jul 28 18:26:33.140070 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:33.139981    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2" event={"ID":"8ba0d2a1-5796-4a6a-8367-f72f1b197613","Type":"ContainerStarted","Data":"31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd"}
Jul 28 18:26:33.140070 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:33.140032    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2" event={"ID":"8ba0d2a1-5796-4a6a-8367-f72f1b197613","Type":"ContainerStarted","Data":"098f07cea369cd158f442c70542a1d4baa6fed7dbd798899e94d8af979af15ec"}
Jul 28 18:26:33.158643 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:33.158572    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2" podStartSLOduration=1.158556089 podStartE2EDuration="1.158556089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:26:33.157507065 +0000 UTC m=+4581.386706184" watchObservedRunningTime="2026-07-28 18:26:33.158556089 +0000 UTC m=+4581.387755184"
Jul 28 18:26:33.831852 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:33.831806    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:42.831604 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:42.831568    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:42.844420 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:42.844383    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:26:43.200472 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:26:43.200394    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:27:27.553623 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:27.553587    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"]
Jul 28 18:27:27.562973 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:27.562943    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"]
Jul 28 18:27:39.310930 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.310896    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"]
Jul 28 18:27:39.313736 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.313716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.316560 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.316538    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"openshift-service-ca.crt\""
Jul 28 18:27:39.316676 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.316579    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"kube-root-ca.crt\""
Jul 28 18:27:39.317640 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.317620    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"default-dockercfg-7pct8\""
Jul 28 18:27:39.317703 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.317627    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-delete-at-nonzero-weight-6b4017d2\"/\"delw-v2-kserve-self-signed-certs\""
Jul 28 18:27:39.326442 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.326420    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"]
Jul 28 18:27:39.360565 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.360534    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.360720 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.360571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw6w8\" (UniqueName: \"kubernetes.io/projected/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-kube-api-access-qw6w8\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.360720 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.360663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.360720 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.360690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.360878 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.360719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-home\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.360878 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.360820    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.461800 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.461761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.461800 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.461806    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw6w8\" (UniqueName: \"kubernetes.io/projected/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-kube-api-access-qw6w8\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.462051 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.461866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.462051 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.461900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.462051 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.461928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-home\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.462051 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.461967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.462352 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.462333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-model-cache\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.462423 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.462389    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tmp-dir\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.462472 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.462449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-home\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.464399 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.464372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-dshm\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.464671 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.464654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tls-certs\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.470115 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.470092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw6w8\" (UniqueName: \"kubernetes.io/projected/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-kube-api-access-qw6w8\") pod \"delw-v2-kserve-5b84cc7ddf-mfspd\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") " pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.624489 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.624458    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:39.758175 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:39.758146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"]
Jul 28 18:27:39.760391 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:27:39.760364    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b7c718d_8b44_4c4c_bf42_bd861e8db53f.slice/crio-5ddf95cbea83e4f3c860c9e6db7e73bbb862360eef788666f268575e6b613e66 WatchSource:0}: Error finding container 5ddf95cbea83e4f3c860c9e6db7e73bbb862360eef788666f268575e6b613e66: Status 404 returned error can't find the container with id 5ddf95cbea83e4f3c860c9e6db7e73bbb862360eef788666f268575e6b613e66
Jul 28 18:27:40.476749 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:40.476711    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd" event={"ID":"9b7c718d-8b44-4c4c-bf42-bd861e8db53f","Type":"ContainerStarted","Data":"4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3"}
Jul 28 18:27:40.477203 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:40.476772    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd" event={"ID":"9b7c718d-8b44-4c4c-bf42-bd861e8db53f","Type":"ContainerStarted","Data":"5ddf95cbea83e4f3c860c9e6db7e73bbb862360eef788666f268575e6b613e66"}
Jul 28 18:27:40.495327 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:40.495267    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd" podStartSLOduration=1.4952472110000001 podStartE2EDuration="1.495247211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:27:40.494463391 +0000 UTC m=+4648.723662489" watchObservedRunningTime="2026-07-28 18:27:40.495247211 +0000 UTC m=+4648.724446309"
Jul 28 18:27:40.625246 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:40.625211    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:42.573640 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.573601    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn" podUID="736e9978-796f-4d00-8366-bde5f348e2ad" containerName="main" containerID="cri-o://94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196" gracePeriod=45
Jul 28 18:27:42.581453 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.581420    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2" podUID="8ba0d2a1-5796-4a6a-8367-f72f1b197613" containerName="main" containerID="cri-o://31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd" gracePeriod=45
Jul 28 18:27:42.852548 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.852524    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:27:42.855297 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.855271    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:27:42.891322 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891288    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-dshm\") pod \"736e9978-796f-4d00-8366-bde5f348e2ad\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") "
Jul 28 18:27:42.891505 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891349    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-home\") pod \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") "
Jul 28 18:27:42.891505 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891379    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-model-cache\") pod \"736e9978-796f-4d00-8366-bde5f348e2ad\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") "
Jul 28 18:27:42.891505 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891427    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vq4d\" (UniqueName: \"kubernetes.io/projected/736e9978-796f-4d00-8366-bde5f348e2ad-kube-api-access-5vq4d\") pod \"736e9978-796f-4d00-8366-bde5f348e2ad\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") "
Jul 28 18:27:42.891707 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891513    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736e9978-796f-4d00-8366-bde5f348e2ad-tls-certs\") pod \"736e9978-796f-4d00-8366-bde5f348e2ad\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") "
Jul 28 18:27:42.891707 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891540    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-model-cache\") pod \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") "
Jul 28 18:27:42.891707 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891570    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-dshm\") pod \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") "
Jul 28 18:27:42.891707 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891599    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-home\") pod \"736e9978-796f-4d00-8366-bde5f348e2ad\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") "
Jul 28 18:27:42.891707 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-home" (OuterVolumeSpecName: "home") pod "8ba0d2a1-5796-4a6a-8367-f72f1b197613" (UID: "8ba0d2a1-5796-4a6a-8367-f72f1b197613"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:27:42.891707 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891648    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xrvq\" (UniqueName: \"kubernetes.io/projected/8ba0d2a1-5796-4a6a-8367-f72f1b197613-kube-api-access-2xrvq\") pod \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") "
Jul 28 18:27:42.891707 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891686    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tmp-dir\") pod \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") "
Jul 28 18:27:42.891707 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891710    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tls-certs\") pod \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\" (UID: \"8ba0d2a1-5796-4a6a-8367-f72f1b197613\") "
Jul 28 18:27:42.892166 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.892047    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ba0d2a1-5796-4a6a-8367-f72f1b197613" (UID: "8ba0d2a1-5796-4a6a-8367-f72f1b197613"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:27:42.892166 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.892054    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ba0d2a1-5796-4a6a-8367-f72f1b197613" (UID: "8ba0d2a1-5796-4a6a-8367-f72f1b197613"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:27:42.892708 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.892394    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-home" (OuterVolumeSpecName: "home") pod "736e9978-796f-4d00-8366-bde5f348e2ad" (UID: "736e9978-796f-4d00-8366-bde5f348e2ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:27:42.892708 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.892451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "736e9978-796f-4d00-8366-bde5f348e2ad" (UID: "736e9978-796f-4d00-8366-bde5f348e2ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:27:42.893225 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.891764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-tmp-dir\") pod \"736e9978-796f-4d00-8366-bde5f348e2ad\" (UID: \"736e9978-796f-4d00-8366-bde5f348e2ad\") "
Jul 28 18:27:42.893540 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.893511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "736e9978-796f-4d00-8366-bde5f348e2ad" (UID: "736e9978-796f-4d00-8366-bde5f348e2ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:27:42.893879 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.893861    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.893954 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.893884    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.893954 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.893896    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.893954 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.893908    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.893954 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.893920    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.893954 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.893932    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.895551 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.895037    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736e9978-796f-4d00-8366-bde5f348e2ad-kube-api-access-5vq4d" (OuterVolumeSpecName: "kube-api-access-5vq4d") pod "736e9978-796f-4d00-8366-bde5f348e2ad" (UID: "736e9978-796f-4d00-8366-bde5f348e2ad"). InnerVolumeSpecName "kube-api-access-5vq4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:27:42.897917 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.897784    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736e9978-796f-4d00-8366-bde5f348e2ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "736e9978-796f-4d00-8366-bde5f348e2ad" (UID: "736e9978-796f-4d00-8366-bde5f348e2ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:27:42.897917 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.897807    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-dshm" (OuterVolumeSpecName: "dshm") pod "736e9978-796f-4d00-8366-bde5f348e2ad" (UID: "736e9978-796f-4d00-8366-bde5f348e2ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:27:42.898152 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.897895    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-dshm" (OuterVolumeSpecName: "dshm") pod "8ba0d2a1-5796-4a6a-8367-f72f1b197613" (UID: "8ba0d2a1-5796-4a6a-8367-f72f1b197613"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:27:42.898253 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.898229    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba0d2a1-5796-4a6a-8367-f72f1b197613-kube-api-access-2xrvq" (OuterVolumeSpecName: "kube-api-access-2xrvq") pod "8ba0d2a1-5796-4a6a-8367-f72f1b197613" (UID: "8ba0d2a1-5796-4a6a-8367-f72f1b197613"). InnerVolumeSpecName "kube-api-access-2xrvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:27:42.899910 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.899888    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ba0d2a1-5796-4a6a-8367-f72f1b197613" (UID: "8ba0d2a1-5796-4a6a-8367-f72f1b197613"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:27:42.994736 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.994701    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xrvq\" (UniqueName: \"kubernetes.io/projected/8ba0d2a1-5796-4a6a-8367-f72f1b197613-kube-api-access-2xrvq\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.994736 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.994732    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ba0d2a1-5796-4a6a-8367-f72f1b197613-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.994923 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.994746    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/736e9978-796f-4d00-8366-bde5f348e2ad-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.994923 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.994758    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vq4d\" (UniqueName: \"kubernetes.io/projected/736e9978-796f-4d00-8366-bde5f348e2ad-kube-api-access-5vq4d\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.994923 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.994769    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/736e9978-796f-4d00-8366-bde5f348e2ad-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:42.994923 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:42.994780    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ba0d2a1-5796-4a6a-8367-f72f1b197613-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:27:43.492655 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.492619    2579 generic.go:356] "Generic (PLEG): container finished" podID="8ba0d2a1-5796-4a6a-8367-f72f1b197613" containerID="31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd" exitCode=0
Jul 28 18:27:43.492834 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.492672    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"
Jul 28 18:27:43.492834 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.492710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2" event={"ID":"8ba0d2a1-5796-4a6a-8367-f72f1b197613","Type":"ContainerDied","Data":"31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd"}
Jul 28 18:27:43.492834 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.492790    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2" event={"ID":"8ba0d2a1-5796-4a6a-8367-f72f1b197613","Type":"ContainerDied","Data":"098f07cea369cd158f442c70542a1d4baa6fed7dbd798899e94d8af979af15ec"}
Jul 28 18:27:43.492834 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.492818    2579 scope.go:117] "RemoveContainer" containerID="31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd"
Jul 28 18:27:43.497577 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.496815    2579 generic.go:356] "Generic (PLEG): container finished" podID="736e9978-796f-4d00-8366-bde5f348e2ad" containerID="94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196" exitCode=0
Jul 28 18:27:43.497577 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.496847    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn" event={"ID":"736e9978-796f-4d00-8366-bde5f348e2ad","Type":"ContainerDied","Data":"94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196"}
Jul 28 18:27:43.497577 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.496884    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn" event={"ID":"736e9978-796f-4d00-8366-bde5f348e2ad","Type":"ContainerDied","Data":"bac21b0d41af50b34de43f875f202a1c1f37d4651ecdbdcd925bf5b6a318a27b"}
Jul 28 18:27:43.497577 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.496944    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"
Jul 28 18:27:43.503207 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.503187    2579 scope.go:117] "RemoveContainer" containerID="31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd"
Jul 28 18:27:43.503456 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:27:43.503439    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd\": container with ID starting with 31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd not found: ID does not exist" containerID="31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd"
Jul 28 18:27:43.503514 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.503464    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd"} err="failed to get container status \"31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd\": rpc error: code = NotFound desc = could not find container \"31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd\": container with ID starting with 31e774f4e8f9e36549b1f185454cd2868bd755becad0f7bccaf2c79191b1b7bd not found: ID does not exist"
Jul 28 18:27:43.503514 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.503482    2579 scope.go:117] "RemoveContainer" containerID="94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196"
Jul 28 18:27:43.512198 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.512182    2579 scope.go:117] "RemoveContainer" containerID="94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196"
Jul 28 18:27:43.512458 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:27:43.512436    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196\": container with ID starting with 94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196 not found: ID does not exist" containerID="94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196"
Jul 28 18:27:43.512530 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.512469    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196"} err="failed to get container status \"94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196\": rpc error: code = NotFound desc = could not find container \"94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196\": container with ID starting with 94703d8a1361a956be7f2c885f237562d50b8e3e6288c35ccaa7ba0a422cc196 not found: ID does not exist"
Jul 28 18:27:43.519326 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.519303    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"]
Jul 28 18:27:43.523687 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.523665    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v2-kserve-86766546f-xznr2"]
Jul 28 18:27:43.534615 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.534583    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"]
Jul 28 18:27:43.536300 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:43.536279    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zqssn"]
Jul 28 18:27:44.320481 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:44.320447    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="736e9978-796f-4d00-8366-bde5f348e2ad" path="/var/lib/kubelet/pods/736e9978-796f-4d00-8366-bde5f348e2ad/volumes"
Jul 28 18:27:44.320867 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:44.320843    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba0d2a1-5796-4a6a-8367-f72f1b197613" path="/var/lib/kubelet/pods/8ba0d2a1-5796-4a6a-8367-f72f1b197613/volumes"
Jul 28 18:27:49.625119 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:49.625064    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:49.637812 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:49.637785    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:27:50.543840 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:27:50.543802    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:28:13.289939 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:13.289858    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"]
Jul 28 18:28:28.309230 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.309166    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd" podUID="9b7c718d-8b44-4c4c-bf42-bd861e8db53f" containerName="main" containerID="cri-o://4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3" gracePeriod=45
Jul 28 18:28:28.561919 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.561863    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:28:28.585722 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.585695    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-model-cache\") pod \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") "
Jul 28 18:28:28.585868 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.585769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tls-certs\") pod \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") "
Jul 28 18:28:28.585868 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.585807    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qw6w8\" (UniqueName: \"kubernetes.io/projected/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-kube-api-access-qw6w8\") pod \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") "
Jul 28 18:28:28.585868 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.585842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-dshm\") pod \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") "
Jul 28 18:28:28.586036 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.585885    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tmp-dir\") pod \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") "
Jul 28 18:28:28.586036 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.585913    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-home\") pod \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\" (UID: \"9b7c718d-8b44-4c4c-bf42-bd861e8db53f\") "
Jul 28 18:28:28.586036 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.585972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-model-cache" (OuterVolumeSpecName: "model-cache") pod "9b7c718d-8b44-4c4c-bf42-bd861e8db53f" (UID: "9b7c718d-8b44-4c4c-bf42-bd861e8db53f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:28:28.586477 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.586260    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:28:28.586586 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.586498    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9b7c718d-8b44-4c4c-bf42-bd861e8db53f" (UID: "9b7c718d-8b44-4c4c-bf42-bd861e8db53f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:28:28.586730 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.586704    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-home" (OuterVolumeSpecName: "home") pod "9b7c718d-8b44-4c4c-bf42-bd861e8db53f" (UID: "9b7c718d-8b44-4c4c-bf42-bd861e8db53f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:28:28.588249 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.588223    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9b7c718d-8b44-4c4c-bf42-bd861e8db53f" (UID: "9b7c718d-8b44-4c4c-bf42-bd861e8db53f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:28:28.588249 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.588238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-dshm" (OuterVolumeSpecName: "dshm") pod "9b7c718d-8b44-4c4c-bf42-bd861e8db53f" (UID: "9b7c718d-8b44-4c4c-bf42-bd861e8db53f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:28:28.588431 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.588411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-kube-api-access-qw6w8" (OuterVolumeSpecName: "kube-api-access-qw6w8") pod "9b7c718d-8b44-4c4c-bf42-bd861e8db53f" (UID: "9b7c718d-8b44-4c4c-bf42-bd861e8db53f"). InnerVolumeSpecName "kube-api-access-qw6w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:28:28.686834 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.686805    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw6w8\" (UniqueName: \"kubernetes.io/projected/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-kube-api-access-qw6w8\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:28:28.686834 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.686832    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:28:28.686997 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.686859    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:28:28.686997 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.686867    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:28:28.686997 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.686875    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b7c718d-8b44-4c4c-bf42-bd861e8db53f-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:28:28.715294 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.715266    2579 generic.go:356] "Generic (PLEG): container finished" podID="9b7c718d-8b44-4c4c-bf42-bd861e8db53f" containerID="4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3" exitCode=0
Jul 28 18:28:28.715412 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.715314    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"
Jul 28 18:28:28.715412 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.715355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd" event={"ID":"9b7c718d-8b44-4c4c-bf42-bd861e8db53f","Type":"ContainerDied","Data":"4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3"}
Jul 28 18:28:28.715412 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.715406    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd" event={"ID":"9b7c718d-8b44-4c4c-bf42-bd861e8db53f","Type":"ContainerDied","Data":"5ddf95cbea83e4f3c860c9e6db7e73bbb862360eef788666f268575e6b613e66"}
Jul 28 18:28:28.715521 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.715437    2579 scope.go:117] "RemoveContainer" containerID="4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3"
Jul 28 18:28:28.724188 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.724168    2579 scope.go:117] "RemoveContainer" containerID="4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3"
Jul 28 18:28:28.724433 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:28:28.724413    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3\": container with ID starting with 4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3 not found: ID does not exist" containerID="4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3"
Jul 28 18:28:28.724479 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.724443    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3"} err="failed to get container status \"4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3\": rpc error: code = NotFound desc = could not find container \"4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3\": container with ID starting with 4ea61370fbf7d529ee318dda8422b7d7cc859b1e6e5f2004e53973eb73822bf3 not found: ID does not exist"
Jul 28 18:28:28.737307 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.737280    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"]
Jul 28 18:28:28.741195 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:28.741173    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd"]
Jul 28 18:28:29.533181 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:29.533144    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-delete-at-nonzero-weight-6b4017d2/delw-v2-kserve-5b84cc7ddf-mfspd" podUID="9b7c718d-8b44-4c4c-bf42-bd861e8db53f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 18:28:30.320247 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:30.320215    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b7c718d-8b44-4c4c-bf42-bd861e8db53f" path="/var/lib/kubelet/pods/9b7c718d-8b44-4c4c-bf42-bd861e8db53f/volumes"
Jul 28 18:28:37.903262 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.903215    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"]
Jul 28 18:28:37.903758 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.903616    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b7c718d-8b44-4c4c-bf42-bd861e8db53f" containerName="main"
Jul 28 18:28:37.903758 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.903628    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b7c718d-8b44-4c4c-bf42-bd861e8db53f" containerName="main"
Jul 28 18:28:37.903758 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.903646    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba0d2a1-5796-4a6a-8367-f72f1b197613" containerName="main"
Jul 28 18:28:37.903758 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.903651    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba0d2a1-5796-4a6a-8367-f72f1b197613" containerName="main"
Jul 28 18:28:37.903758 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.903661    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="736e9978-796f-4d00-8366-bde5f348e2ad" containerName="main"
Jul 28 18:28:37.903758 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.903667    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="736e9978-796f-4d00-8366-bde5f348e2ad" containerName="main"
Jul 28 18:28:37.903758 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.903728    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="736e9978-796f-4d00-8366-bde5f348e2ad" containerName="main"
Jul 28 18:28:37.903758 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.903735    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b7c718d-8b44-4c4c-bf42-bd861e8db53f" containerName="main"
Jul 28 18:28:37.903758 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.903742    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba0d2a1-5796-4a6a-8367-f72f1b197613" containerName="main"
Jul 28 18:28:37.906937 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.906910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:37.909848 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.909827    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Jul 28 18:28:37.909848 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.909835    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Jul 28 18:28:37.910942 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.910923    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-6b8kj\""
Jul 28 18:28:37.911040 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.910965    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v1-kserve-self-signed-certs\""
Jul 28 18:28:37.915972 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.915949    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"]
Jul 28 18:28:37.967900 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.967873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:37.968021 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.967905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:37.968021 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.967925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a66541d-150e-4ad4-8c91-5beab5a663f9-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:37.968021 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.967951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-home\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:37.968021 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.967974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-dshm\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:37.968201 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:37.968024    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bv79\" (UniqueName: \"kubernetes.io/projected/6a66541d-150e-4ad4-8c91-5beab5a663f9-kube-api-access-7bv79\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.068748 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.068719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-home\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.068907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.068752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-dshm\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.068907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.068802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bv79\" (UniqueName: \"kubernetes.io/projected/6a66541d-150e-4ad4-8c91-5beab5a663f9-kube-api-access-7bv79\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.068907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.068867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.068907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.068896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.069160 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.068922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a66541d-150e-4ad4-8c91-5beab5a663f9-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.069228 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.069210    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-home\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.069327 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.069305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-tmp-dir\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.069383 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.069333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-model-cache\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.071103 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.071050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-dshm\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.071434 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.071415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a66541d-150e-4ad4-8c91-5beab5a663f9-tls-certs\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.077571 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.077552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bv79\" (UniqueName: \"kubernetes.io/projected/6a66541d-150e-4ad4-8c91-5beab5a663f9-kube-api-access-7bv79\") pod \"roll-v1-kserve-69cf8bcfc-cdx8r\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") " pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.218468 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.218377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:38.357704 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.357519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"]
Jul 28 18:28:38.360172 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:28:38.360138    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a66541d_150e_4ad4_8c91_5beab5a663f9.slice/crio-463d46471b4fb67831382d616f8942e86300044615534deb8e3a7e9ba309c07c WatchSource:0}: Error finding container 463d46471b4fb67831382d616f8942e86300044615534deb8e3a7e9ba309c07c: Status 404 returned error can't find the container with id 463d46471b4fb67831382d616f8942e86300044615534deb8e3a7e9ba309c07c
Jul 28 18:28:38.764200 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.764163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r" event={"ID":"6a66541d-150e-4ad4-8c91-5beab5a663f9","Type":"ContainerStarted","Data":"c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef"}
Jul 28 18:28:38.764395 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.764215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r" event={"ID":"6a66541d-150e-4ad4-8c91-5beab5a663f9","Type":"ContainerStarted","Data":"463d46471b4fb67831382d616f8942e86300044615534deb8e3a7e9ba309c07c"}
Jul 28 18:28:38.784886 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:38.784840    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r" podStartSLOduration=1.784826862 podStartE2EDuration="1.784826862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:28:38.783115519 +0000 UTC m=+4707.012314630" watchObservedRunningTime="2026-07-28 18:28:38.784826862 +0000 UTC m=+4707.014025957"
Jul 28 18:28:39.219257 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:39.219217    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:43.003502 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.003467    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"]
Jul 28 18:28:43.011138 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.011096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.014280 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.014256    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Jul 28 18:28:43.018982 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.018959    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"]
Jul 28 18:28:43.114983 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.114951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-home\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.115209 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.114994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-dshm\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.115209 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.115010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.115209 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.115049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnbx\" (UniqueName: \"kubernetes.io/projected/523df82c-50a2-4c4a-b4b5-d0156e1896d5-kube-api-access-6bnbx\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.115209 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.115126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.115209 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.115164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.216404 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.216357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-dshm\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.216404 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.216404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.216604 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.216517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnbx\" (UniqueName: \"kubernetes.io/projected/523df82c-50a2-4c4a-b4b5-d0156e1896d5-kube-api-access-6bnbx\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.216604 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.216571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.216776 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.216743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.216924 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.216865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-home\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.216924 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.216871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.217071 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.217049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.217212 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.217196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-home\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.219040 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.219018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-dshm\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.219283 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.219267    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.225011 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.224991    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnbx\" (UniqueName: \"kubernetes.io/projected/523df82c-50a2-4c4a-b4b5-d0156e1896d5-kube-api-access-6bnbx\") pod \"roll-v2-kserve-d4dbf9d67-tsg9n\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.324320 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.324293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:43.457562 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.457532    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"]
Jul 28 18:28:43.459136 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:28:43.459108    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523df82c_50a2_4c4a_b4b5_d0156e1896d5.slice/crio-04f6ed1f85ec3f3f6cf43fbfa28cab581032e3c50c8984d0030e66f3ecf44fc4 WatchSource:0}: Error finding container 04f6ed1f85ec3f3f6cf43fbfa28cab581032e3c50c8984d0030e66f3ecf44fc4: Status 404 returned error can't find the container with id 04f6ed1f85ec3f3f6cf43fbfa28cab581032e3c50c8984d0030e66f3ecf44fc4
Jul 28 18:28:43.789495 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.789415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n" event={"ID":"523df82c-50a2-4c4a-b4b5-d0156e1896d5","Type":"ContainerStarted","Data":"aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812"}
Jul 28 18:28:43.789495 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.789470    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n" event={"ID":"523df82c-50a2-4c4a-b4b5-d0156e1896d5","Type":"ContainerStarted","Data":"04f6ed1f85ec3f3f6cf43fbfa28cab581032e3c50c8984d0030e66f3ecf44fc4"}
Jul 28 18:28:43.820636 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:43.820589    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n" podStartSLOduration=1.820573527 podStartE2EDuration="1.820573527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:28:43.818311362 +0000 UTC m=+4712.047510461" watchObservedRunningTime="2026-07-28 18:28:43.820573527 +0000 UTC m=+4712.049772622"
Jul 28 18:28:44.325091 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:44.325057    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:48.218749 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:48.218718    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:48.231435 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:48.231409    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:48.825196 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:48.825163    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:28:53.324837 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:53.324793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:53.337663 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:53.337631    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:28:53.854309 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:28:53.854273    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:30:10.838483 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:10.838443    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"]
Jul 28 18:30:10.843387 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:10.843348    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"]
Jul 28 18:30:12.112430 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.112396    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"]
Jul 28 18:30:12.115937 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.115917    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.118531 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.118514    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Jul 28 18:30:12.119952 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.119920    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Jul 28 18:30:12.119952 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.119919    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-v29pc\""
Jul 28 18:30:12.120151 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.119977    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Jul 28 18:30:12.128552 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.128531    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"]
Jul 28 18:30:12.200621 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.200575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssc28\" (UniqueName: \"kubernetes.io/projected/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-kube-api-access-ssc28\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.200621 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.200611    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.200818 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.200635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-dshm\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.200818 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.200682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-home\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.200818 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.200703    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-model-cache\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.200818 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.200719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.302281 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.302243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssc28\" (UniqueName: \"kubernetes.io/projected/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-kube-api-access-ssc28\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.302474 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.302289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.302474 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.302307    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-dshm\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.302474 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.302323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-home\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.302474 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.302342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-model-cache\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.302474 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.302360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.302757 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.302710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.302803 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.302775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-home\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.302803 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.302792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-model-cache\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.304562 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.304538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-dshm\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.305301 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.305286    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Jul 28 18:30:12.311544 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.311526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Jul 28 18:30:12.314993 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.314972    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.321888 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.321868    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Jul 28 18:30:12.332227 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.332208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssc28\" (UniqueName: \"kubernetes.io/projected/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-kube-api-access-ssc28\") pod \"owner-v1-kserve-75579ff5bc-c5g66\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.428739 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.428665    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-v29pc\""
Jul 28 18:30:12.436873 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.436853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:12.574445 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.574423    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"]
Jul 28 18:30:12.576482 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:30:12.576454    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d0aa0c_5de5_4fb7_abd2_a219ea92c159.slice/crio-704ed75e847adafb60ea817a2dd1d873153127d139479dafc932b45aadcd4ac4 WatchSource:0}: Error finding container 704ed75e847adafb60ea817a2dd1d873153127d139479dafc932b45aadcd4ac4: Status 404 returned error can't find the container with id 704ed75e847adafb60ea817a2dd1d873153127d139479dafc932b45aadcd4ac4
Jul 28 18:30:12.656971 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.656951    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:30:12.686147 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:12.686095    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:30:13.240103 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:13.240054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66" event={"ID":"24d0aa0c-5de5-4fb7-abd2-a219ea92c159","Type":"ContainerStarted","Data":"3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405"}
Jul 28 18:30:13.240481 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:13.240119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66" event={"ID":"24d0aa0c-5de5-4fb7-abd2-a219ea92c159","Type":"ContainerStarted","Data":"704ed75e847adafb60ea817a2dd1d873153127d139479dafc932b45aadcd4ac4"}
Jul 28 18:30:13.260687 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:13.260634    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66" podStartSLOduration=1.260621245 podStartE2EDuration="1.260621245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:13.259936389 +0000 UTC m=+4801.489135486" watchObservedRunningTime="2026-07-28 18:30:13.260621245 +0000 UTC m=+4801.489820341"
Jul 28 18:30:13.438005 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:13.437975    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:22.437264 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:22.437232    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:22.449676 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:22.449651    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:23.305398 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:23.305362    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:30:25.858849 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:25.858813    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r" podUID="6a66541d-150e-4ad4-8c91-5beab5a663f9" containerName="main" containerID="cri-o://c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef" gracePeriod=45
Jul 28 18:30:25.863184 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:25.863140    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n" podUID="523df82c-50a2-4c4a-b4b5-d0156e1896d5" containerName="main" containerID="cri-o://aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812" gracePeriod=45
Jul 28 18:30:26.116319 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.116265    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:30:26.138787 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.138765    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:30:26.234873 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.234840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-tmp-dir\") pod \"6a66541d-150e-4ad4-8c91-5beab5a663f9\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") "
Jul 28 18:30:26.234873 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.234878    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-dshm\") pod \"6a66541d-150e-4ad4-8c91-5beab5a663f9\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") "
Jul 28 18:30:26.235167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.234901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a66541d-150e-4ad4-8c91-5beab5a663f9-tls-certs\") pod \"6a66541d-150e-4ad4-8c91-5beab5a663f9\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") "
Jul 28 18:30:26.235167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.234928    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-dshm\") pod \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") "
Jul 28 18:30:26.235167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.234958    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-model-cache\") pod \"6a66541d-150e-4ad4-8c91-5beab5a663f9\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") "
Jul 28 18:30:26.235167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235001    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-home\") pod \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") "
Jul 28 18:30:26.235167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235037    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bv79\" (UniqueName: \"kubernetes.io/projected/6a66541d-150e-4ad4-8c91-5beab5a663f9-kube-api-access-7bv79\") pod \"6a66541d-150e-4ad4-8c91-5beab5a663f9\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") "
Jul 28 18:30:26.235167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235073    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bnbx\" (UniqueName: \"kubernetes.io/projected/523df82c-50a2-4c4a-b4b5-d0156e1896d5-kube-api-access-6bnbx\") pod \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") "
Jul 28 18:30:26.235167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-home\") pod \"6a66541d-150e-4ad4-8c91-5beab5a663f9\" (UID: \"6a66541d-150e-4ad4-8c91-5beab5a663f9\") "
Jul 28 18:30:26.235167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235166    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tls-certs\") pod \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") "
Jul 28 18:30:26.235640 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235182    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a66541d-150e-4ad4-8c91-5beab5a663f9" (UID: "6a66541d-150e-4ad4-8c91-5beab5a663f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:26.235640 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235215    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tmp-dir\") pod \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") "
Jul 28 18:30:26.235640 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235260    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-model-cache\") pod \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\" (UID: \"523df82c-50a2-4c4a-b4b5-d0156e1896d5\") "
Jul 28 18:30:26.235640 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235548    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a66541d-150e-4ad4-8c91-5beab5a663f9" (UID: "6a66541d-150e-4ad4-8c91-5beab5a663f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:26.235871 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235666    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-home" (OuterVolumeSpecName: "home") pod "523df82c-50a2-4c4a-b4b5-d0156e1896d5" (UID: "523df82c-50a2-4c4a-b4b5-d0156e1896d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:26.235871 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235683    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.235871 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235703    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.235871 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "523df82c-50a2-4c4a-b4b5-d0156e1896d5" (UID: "523df82c-50a2-4c4a-b4b5-d0156e1896d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:26.236102 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.235977    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "523df82c-50a2-4c4a-b4b5-d0156e1896d5" (UID: "523df82c-50a2-4c4a-b4b5-d0156e1896d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:26.236102 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.236019    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-home" (OuterVolumeSpecName: "home") pod "6a66541d-150e-4ad4-8c91-5beab5a663f9" (UID: "6a66541d-150e-4ad4-8c91-5beab5a663f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:26.237953 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.237921    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-dshm" (OuterVolumeSpecName: "dshm") pod "6a66541d-150e-4ad4-8c91-5beab5a663f9" (UID: "6a66541d-150e-4ad4-8c91-5beab5a663f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:26.238100 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.237975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523df82c-50a2-4c4a-b4b5-d0156e1896d5-kube-api-access-6bnbx" (OuterVolumeSpecName: "kube-api-access-6bnbx") pod "523df82c-50a2-4c4a-b4b5-d0156e1896d5" (UID: "523df82c-50a2-4c4a-b4b5-d0156e1896d5"). InnerVolumeSpecName "kube-api-access-6bnbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:26.238100 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.238001    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a66541d-150e-4ad4-8c91-5beab5a663f9-kube-api-access-7bv79" (OuterVolumeSpecName: "kube-api-access-7bv79") pod "6a66541d-150e-4ad4-8c91-5beab5a663f9" (UID: "6a66541d-150e-4ad4-8c91-5beab5a663f9"). InnerVolumeSpecName "kube-api-access-7bv79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:30:26.238290 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.238266    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-dshm" (OuterVolumeSpecName: "dshm") pod "523df82c-50a2-4c4a-b4b5-d0156e1896d5" (UID: "523df82c-50a2-4c4a-b4b5-d0156e1896d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:30:26.238414 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.238343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "523df82c-50a2-4c4a-b4b5-d0156e1896d5" (UID: "523df82c-50a2-4c4a-b4b5-d0156e1896d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:30:26.239123 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.239062    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a66541d-150e-4ad4-8c91-5beab5a663f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a66541d-150e-4ad4-8c91-5beab5a663f9" (UID: "6a66541d-150e-4ad4-8c91-5beab5a663f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:30:26.310924 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.310888    2579 generic.go:356] "Generic (PLEG): container finished" podID="523df82c-50a2-4c4a-b4b5-d0156e1896d5" containerID="aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812" exitCode=0
Jul 28 18:30:26.311106 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.310932    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"
Jul 28 18:30:26.311106 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.310966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n" event={"ID":"523df82c-50a2-4c4a-b4b5-d0156e1896d5","Type":"ContainerDied","Data":"aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812"}
Jul 28 18:30:26.311106 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.311020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n" event={"ID":"523df82c-50a2-4c4a-b4b5-d0156e1896d5","Type":"ContainerDied","Data":"04f6ed1f85ec3f3f6cf43fbfa28cab581032e3c50c8984d0030e66f3ecf44fc4"}
Jul 28 18:30:26.311106 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.311054    2579 scope.go:117] "RemoveContainer" containerID="aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812"
Jul 28 18:30:26.312436 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.312384    2579 generic.go:356] "Generic (PLEG): container finished" podID="6a66541d-150e-4ad4-8c91-5beab5a663f9" containerID="c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef" exitCode=0
Jul 28 18:30:26.312436 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.312431    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"
Jul 28 18:30:26.312619 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.312469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r" event={"ID":"6a66541d-150e-4ad4-8c91-5beab5a663f9","Type":"ContainerDied","Data":"c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef"}
Jul 28 18:30:26.312619 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.312522    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r" event={"ID":"6a66541d-150e-4ad4-8c91-5beab5a663f9","Type":"ContainerDied","Data":"463d46471b4fb67831382d616f8942e86300044615534deb8e3a7e9ba309c07c"}
Jul 28 18:30:26.320637 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.320617    2579 scope.go:117] "RemoveContainer" containerID="aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812"
Jul 28 18:30:26.320914 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:30:26.320894    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812\": container with ID starting with aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812 not found: ID does not exist" containerID="aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812"
Jul 28 18:30:26.320979 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.320923    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812"} err="failed to get container status \"aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812\": rpc error: code = NotFound desc = could not find container \"aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812\": container with ID starting with aac940c33074f76284d3c8bffc7b79ab0511d1f5e410a68a15678c32e18f1812 not found: ID does not exist"
Jul 28 18:30:26.320979 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.320941    2579 scope.go:117] "RemoveContainer" containerID="c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef"
Jul 28 18:30:26.339176 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339149    2579 scope.go:117] "RemoveContainer" containerID="c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef"
Jul 28 18:30:26.339272 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339165    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.339272 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339253    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bv79\" (UniqueName: \"kubernetes.io/projected/6a66541d-150e-4ad4-8c91-5beab5a663f9-kube-api-access-7bv79\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.339272 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339268    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bnbx\" (UniqueName: \"kubernetes.io/projected/523df82c-50a2-4c4a-b4b5-d0156e1896d5-kube-api-access-6bnbx\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.339389 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339283    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.339389 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339296    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.339389 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339329    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.339389 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339341    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.339389 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339352    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a66541d-150e-4ad4-8c91-5beab5a663f9-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.339389 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339363    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a66541d-150e-4ad4-8c91-5beab5a663f9-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.339629 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339381    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/523df82c-50a2-4c4a-b4b5-d0156e1896d5-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:30:26.339629 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:30:26.339581    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef\": container with ID starting with c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef not found: ID does not exist" containerID="c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef"
Jul 28 18:30:26.339629 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.339604    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef"} err="failed to get container status \"c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef\": rpc error: code = NotFound desc = could not find container \"c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef\": container with ID starting with c882698804ed05a5085253c89cd2afde567843b43d0eaf07fb5c6ece6e61b4ef not found: ID does not exist"
Jul 28 18:30:26.342694 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.342671    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"]
Jul 28 18:30:26.347040 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.347015    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v1-kserve-69cf8bcfc-cdx8r"]
Jul 28 18:30:26.358189 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.358168    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"]
Jul 28 18:30:26.360469 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:26.360447    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-tsg9n"]
Jul 28 18:30:28.320503 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:28.320469    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="523df82c-50a2-4c4a-b4b5-d0156e1896d5" path="/var/lib/kubelet/pods/523df82c-50a2-4c4a-b4b5-d0156e1896d5/volumes"
Jul 28 18:30:28.320919 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:30:28.320889    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a66541d-150e-4ad4-8c91-5beab5a663f9" path="/var/lib/kubelet/pods/6a66541d-150e-4ad4-8c91-5beab5a663f9/volumes"
Jul 28 18:31:11.979015 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:11.978971    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"]
Jul 28 18:31:26.998498 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:26.998454    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66" podUID="24d0aa0c-5de5-4fb7-abd2-a219ea92c159" containerName="main" containerID="cri-o://3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405" gracePeriod=45
Jul 28 18:31:27.253961 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.253903    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:31:27.389595 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.389563    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssc28\" (UniqueName: \"kubernetes.io/projected/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-kube-api-access-ssc28\") pod \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") "
Jul 28 18:31:27.389785 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.389629    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-model-cache\") pod \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") "
Jul 28 18:31:27.389785 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.389655    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tls-certs\") pod \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") "
Jul 28 18:31:27.389785 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.389691    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-home\") pod \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") "
Jul 28 18:31:27.389785 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.389710    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-dshm\") pod \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") "
Jul 28 18:31:27.389785 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.389775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tmp-dir\") pod \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\" (UID: \"24d0aa0c-5de5-4fb7-abd2-a219ea92c159\") "
Jul 28 18:31:27.390110 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.389893    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-model-cache" (OuterVolumeSpecName: "model-cache") pod "24d0aa0c-5de5-4fb7-abd2-a219ea92c159" (UID: "24d0aa0c-5de5-4fb7-abd2-a219ea92c159"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:31:27.390110 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.389930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-home" (OuterVolumeSpecName: "home") pod "24d0aa0c-5de5-4fb7-abd2-a219ea92c159" (UID: "24d0aa0c-5de5-4fb7-abd2-a219ea92c159"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:31:27.390110 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.390101    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "24d0aa0c-5de5-4fb7-abd2-a219ea92c159" (UID: "24d0aa0c-5de5-4fb7-abd2-a219ea92c159"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:31:27.390283 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.390156    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:31:27.390283 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.390171    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:31:27.391978 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.391941    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-dshm" (OuterVolumeSpecName: "dshm") pod "24d0aa0c-5de5-4fb7-abd2-a219ea92c159" (UID: "24d0aa0c-5de5-4fb7-abd2-a219ea92c159"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:31:27.392107 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.391998    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "24d0aa0c-5de5-4fb7-abd2-a219ea92c159" (UID: "24d0aa0c-5de5-4fb7-abd2-a219ea92c159"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:31:27.392107 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.392004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-kube-api-access-ssc28" (OuterVolumeSpecName: "kube-api-access-ssc28") pod "24d0aa0c-5de5-4fb7-abd2-a219ea92c159" (UID: "24d0aa0c-5de5-4fb7-abd2-a219ea92c159"). InnerVolumeSpecName "kube-api-access-ssc28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:31:27.490781 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.490749    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:31:27.490781 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.490776    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssc28\" (UniqueName: \"kubernetes.io/projected/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-kube-api-access-ssc28\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:31:27.490781 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.490787    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:31:27.490993 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.490795    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24d0aa0c-5de5-4fb7-abd2-a219ea92c159-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:31:27.606281 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.606250    2579 generic.go:356] "Generic (PLEG): container finished" podID="24d0aa0c-5de5-4fb7-abd2-a219ea92c159" containerID="3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405" exitCode=0
Jul 28 18:31:27.606440 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.606302    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"
Jul 28 18:31:27.606440 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.606328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66" event={"ID":"24d0aa0c-5de5-4fb7-abd2-a219ea92c159","Type":"ContainerDied","Data":"3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405"}
Jul 28 18:31:27.606440 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.606371    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66" event={"ID":"24d0aa0c-5de5-4fb7-abd2-a219ea92c159","Type":"ContainerDied","Data":"704ed75e847adafb60ea817a2dd1d873153127d139479dafc932b45aadcd4ac4"}
Jul 28 18:31:27.606440 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.606391    2579 scope.go:117] "RemoveContainer" containerID="3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405"
Jul 28 18:31:27.615328 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.615314    2579 scope.go:117] "RemoveContainer" containerID="3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405"
Jul 28 18:31:27.615595 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:31:27.615575    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405\": container with ID starting with 3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405 not found: ID does not exist" containerID="3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405"
Jul 28 18:31:27.615697 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.615602    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405"} err="failed to get container status \"3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405\": rpc error: code = NotFound desc = could not find container \"3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405\": container with ID starting with 3a40eab145b8889022b087ea87973a97b2e3ab1347a89310905192c6b4c61405 not found: ID does not exist"
Jul 28 18:31:27.628351 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.628332    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"]
Jul 28 18:31:27.635256 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:27.635236    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-c5g66"]
Jul 28 18:31:28.319680 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:28.319647    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24d0aa0c-5de5-4fb7-abd2-a219ea92c159" path="/var/lib/kubelet/pods/24d0aa0c-5de5-4fb7-abd2-a219ea92c159/volumes"
Jul 28 18:31:47.484471 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.484437    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"]
Jul 28 18:31:47.486769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.484839    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24d0aa0c-5de5-4fb7-abd2-a219ea92c159" containerName="main"
Jul 28 18:31:47.486769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.484853    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="24d0aa0c-5de5-4fb7-abd2-a219ea92c159" containerName="main"
Jul 28 18:31:47.486769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.484865    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a66541d-150e-4ad4-8c91-5beab5a663f9" containerName="main"
Jul 28 18:31:47.486769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.484871    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a66541d-150e-4ad4-8c91-5beab5a663f9" containerName="main"
Jul 28 18:31:47.486769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.484891    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="523df82c-50a2-4c4a-b4b5-d0156e1896d5" containerName="main"
Jul 28 18:31:47.486769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.484898    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="523df82c-50a2-4c4a-b4b5-d0156e1896d5" containerName="main"
Jul 28 18:31:47.486769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.484968    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a66541d-150e-4ad4-8c91-5beab5a663f9" containerName="main"
Jul 28 18:31:47.486769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.484978    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="523df82c-50a2-4c4a-b4b5-d0156e1896d5" containerName="main"
Jul 28 18:31:47.486769 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.484985    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="24d0aa0c-5de5-4fb7-abd2-a219ea92c159" containerName="main"
Jul 28 18:31:47.487761 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.487744    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.490309 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.490285    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Jul 28 18:31:47.490427 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.490340    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Jul 28 18:31:47.491462 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.491446    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-q8pqb\""
Jul 28 18:31:47.491588 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.491494    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Jul 28 18:31:47.496896 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.496871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"]
Jul 28 18:31:47.552516 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.552476    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.552692 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.552525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.552692 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.552547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.552692 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.552598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5851a3-c682-43de-b138-e15c9f3a572c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.552692 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.552622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlkz5\" (UniqueName: \"kubernetes.io/projected/4e5851a3-c682-43de-b138-e15c9f3a572c-kube-api-access-tlkz5\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.552692 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.552653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.654176 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.654138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.654358 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.654200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.654358 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.654228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.654358 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.654274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5851a3-c682-43de-b138-e15c9f3a572c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.654358 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.654304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlkz5\" (UniqueName: \"kubernetes.io/projected/4e5851a3-c682-43de-b138-e15c9f3a572c-kube-api-access-tlkz5\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.654358 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.654338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.654652 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.654612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.654652 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.654643    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.654833 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.654812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.656484 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.656463    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.657026 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.657005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5851a3-c682-43de-b138-e15c9f3a572c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.663456 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.663431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlkz5\" (UniqueName: \"kubernetes.io/projected/4e5851a3-c682-43de-b138-e15c9f3a572c-kube-api-access-tlkz5\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.798967 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.798882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:47.936499 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:47.936468    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"]
Jul 28 18:31:47.939333 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:31:47.939307    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5851a3_c682_43de_b138_e15c9f3a572c.slice/crio-769a37c3fce26db3edbb508d652839c6bb8ad2163355f166e08bdfd880d66f28 WatchSource:0}: Error finding container 769a37c3fce26db3edbb508d652839c6bb8ad2163355f166e08bdfd880d66f28: Status 404 returned error can't find the container with id 769a37c3fce26db3edbb508d652839c6bb8ad2163355f166e08bdfd880d66f28
Jul 28 18:31:48.715056 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:48.715015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv" event={"ID":"4e5851a3-c682-43de-b138-e15c9f3a572c","Type":"ContainerStarted","Data":"af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948"}
Jul 28 18:31:48.715548 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:48.715070    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv" event={"ID":"4e5851a3-c682-43de-b138-e15c9f3a572c","Type":"ContainerStarted","Data":"769a37c3fce26db3edbb508d652839c6bb8ad2163355f166e08bdfd880d66f28"}
Jul 28 18:31:48.733272 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:48.733223    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv" podStartSLOduration=1.733206529 podStartE2EDuration="1.733206529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:31:48.73182604 +0000 UTC m=+4896.961025137" watchObservedRunningTime="2026-07-28 18:31:48.733206529 +0000 UTC m=+4896.962405618"
Jul 28 18:31:48.799455 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:48.799418    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:57.799032 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:57.798991    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:57.811389 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:57.811354    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:31:58.780766 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:31:58.780670    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:33:33.243890 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.243858    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"]
Jul 28 18:33:33.253231 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:33:33.253200    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 28 18:33:33.253231 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:33:33.253231 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:33:33.253231 ip-10-0-136-111 kubenswrapper[2579]:  > containerID="af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948" cmd=["/bin/sleep","15"]
Jul 28 18:33:33.253484 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.253243    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv" message="" err=<
Jul 28 18:33:33.253484 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:33:33.253484 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:33:33.253484 ip-10-0-136-111 kubenswrapper[2579]:  >
Jul 28 18:33:33.253484 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:33:33.253262    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 28 18:33:33.253484 ip-10-0-136-111 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 28 18:33:33.253484 ip-10-0-136-111 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 28 18:33:33.253484 ip-10-0-136-111 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv" podUID="4e5851a3-c682-43de-b138-e15c9f3a572c" containerName="main" containerID="cri-o://af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948"
Jul 28 18:33:33.253484 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.253295    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv" podUID="4e5851a3-c682-43de-b138-e15c9f3a572c" containerName="main" containerID="cri-o://af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948" gracePeriod=60
Jul 28 18:33:33.610726 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.610697    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:33:33.709403 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.709370    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-home\") pod \"4e5851a3-c682-43de-b138-e15c9f3a572c\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") "
Jul 28 18:33:33.709574 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.709428    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-tmp-dir\") pod \"4e5851a3-c682-43de-b138-e15c9f3a572c\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") "
Jul 28 18:33:33.709574 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.709471    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-dshm\") pod \"4e5851a3-c682-43de-b138-e15c9f3a572c\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") "
Jul 28 18:33:33.709574 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.709542    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-model-cache\") pod \"4e5851a3-c682-43de-b138-e15c9f3a572c\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") "
Jul 28 18:33:33.709730 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.709573    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5851a3-c682-43de-b138-e15c9f3a572c-tls-certs\") pod \"4e5851a3-c682-43de-b138-e15c9f3a572c\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") "
Jul 28 18:33:33.709730 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.709638    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlkz5\" (UniqueName: \"kubernetes.io/projected/4e5851a3-c682-43de-b138-e15c9f3a572c-kube-api-access-tlkz5\") pod \"4e5851a3-c682-43de-b138-e15c9f3a572c\" (UID: \"4e5851a3-c682-43de-b138-e15c9f3a572c\") "
Jul 28 18:33:33.709730 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.709691    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-home" (OuterVolumeSpecName: "home") pod "4e5851a3-c682-43de-b138-e15c9f3a572c" (UID: "4e5851a3-c682-43de-b138-e15c9f3a572c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:33:33.709893 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.709745    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e5851a3-c682-43de-b138-e15c9f3a572c" (UID: "4e5851a3-c682-43de-b138-e15c9f3a572c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:33:33.709893 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.709790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e5851a3-c682-43de-b138-e15c9f3a572c" (UID: "4e5851a3-c682-43de-b138-e15c9f3a572c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:33:33.710053 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.710035    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:33:33.710152 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.710073    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:33:33.710152 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.710113    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:33:33.711852 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.711819    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e5851a3-c682-43de-b138-e15c9f3a572c-kube-api-access-tlkz5" (OuterVolumeSpecName: "kube-api-access-tlkz5") pod "4e5851a3-c682-43de-b138-e15c9f3a572c" (UID: "4e5851a3-c682-43de-b138-e15c9f3a572c"). InnerVolumeSpecName "kube-api-access-tlkz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:33:33.711965 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.711855    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-dshm" (OuterVolumeSpecName: "dshm") pod "4e5851a3-c682-43de-b138-e15c9f3a572c" (UID: "4e5851a3-c682-43de-b138-e15c9f3a572c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:33:33.711965 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.711876    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e5851a3-c682-43de-b138-e15c9f3a572c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e5851a3-c682-43de-b138-e15c9f3a572c" (UID: "4e5851a3-c682-43de-b138-e15c9f3a572c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:33:33.811053 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.810978    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e5851a3-c682-43de-b138-e15c9f3a572c-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:33:33.811053 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.811002    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e5851a3-c682-43de-b138-e15c9f3a572c-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:33:33.811053 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:33.811012    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlkz5\" (UniqueName: \"kubernetes.io/projected/4e5851a3-c682-43de-b138-e15c9f3a572c-kube-api-access-tlkz5\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:33:34.248353 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:34.248321    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e5851a3-c682-43de-b138-e15c9f3a572c" containerID="af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948" exitCode=0
Jul 28 18:33:34.248801 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:34.248369    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"
Jul 28 18:33:34.248801 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:34.248369    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv" event={"ID":"4e5851a3-c682-43de-b138-e15c9f3a572c","Type":"ContainerDied","Data":"af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948"}
Jul 28 18:33:34.248801 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:34.248508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv" event={"ID":"4e5851a3-c682-43de-b138-e15c9f3a572c","Type":"ContainerDied","Data":"769a37c3fce26db3edbb508d652839c6bb8ad2163355f166e08bdfd880d66f28"}
Jul 28 18:33:34.248801 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:34.248542    2579 scope.go:117] "RemoveContainer" containerID="af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948"
Jul 28 18:33:34.259044 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:34.258869    2579 scope.go:117] "RemoveContainer" containerID="af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948"
Jul 28 18:33:34.259136 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:33:34.259116    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948\": container with ID starting with af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948 not found: ID does not exist" containerID="af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948"
Jul 28 18:33:34.259182 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:34.259144    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948"} err="failed to get container status \"af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948\": rpc error: code = NotFound desc = could not find container \"af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948\": container with ID starting with af350d98c8dfad63c14761d6d36086051860c0121b301cde707aa89f2e8b4948 not found: ID does not exist"
Jul 28 18:33:34.270960 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:34.270935    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"]
Jul 28 18:33:34.276837 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:34.276817    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997sjmzv"]
Jul 28 18:33:34.326915 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:34.326878    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e5851a3-c682-43de-b138-e15c9f3a572c" path="/var/lib/kubelet/pods/4e5851a3-c682-43de-b138-e15c9f3a572c/volumes"
Jul 28 18:33:49.899183 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.899152    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"]
Jul 28 18:33:49.899611 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.899565    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e5851a3-c682-43de-b138-e15c9f3a572c" containerName="main"
Jul 28 18:33:49.899611 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.899577    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e5851a3-c682-43de-b138-e15c9f3a572c" containerName="main"
Jul 28 18:33:49.899687 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.899644    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e5851a3-c682-43de-b138-e15c9f3a572c" containerName="main"
Jul 28 18:33:49.902881 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.902858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:49.905524 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.905489    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"seaweedfs-s3-creds\""
Jul 28 18:33:49.905815 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.905797    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Jul 28 18:33:49.905920 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.905803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Jul 28 18:33:49.905920 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.905834    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-k2crv\""
Jul 28 18:33:49.905920 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.905802    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 28 18:33:49.914743 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:49.914721    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"]
Jul 28 18:33:50.054472 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.054424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.054680 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.054485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.054680 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.054510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.054680 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.054566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrkxr\" (UniqueName: \"kubernetes.io/projected/fd44e575-fc79-4f46-bc78-00028b89f4f8-kube-api-access-hrkxr\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.054680 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.054598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.054680 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.054656    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.054680 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.054680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd44e575-fc79-4f46-bc78-00028b89f4f8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156045 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.155956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156045 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.155998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156045 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.156022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkxr\" (UniqueName: \"kubernetes.io/projected/fd44e575-fc79-4f46-bc78-00028b89f4f8-kube-api-access-hrkxr\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156045 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.156046    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156423 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.156153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156423 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.156181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd44e575-fc79-4f46-bc78-00028b89f4f8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156423 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.156218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156582 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.156504    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156582 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.156513    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156582 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.156569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.156689 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.156633    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.158511 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.158490    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.158815 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.158791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd44e575-fc79-4f46-bc78-00028b89f4f8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.165868 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.165845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrkxr\" (UniqueName: \"kubernetes.io/projected/fd44e575-fc79-4f46-bc78-00028b89f4f8-kube-api-access-hrkxr\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") " pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.214412 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.214382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:33:50.356060 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:50.356031    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"]
Jul 28 18:33:50.357508 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:33:50.357477    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd44e575_fc79_4f46_bc78_00028b89f4f8.slice/crio-01f98e234ff893c3bebf86f660bdb39ec9a79f2c7fe813b84f75f006041d8f20 WatchSource:0}: Error finding container 01f98e234ff893c3bebf86f660bdb39ec9a79f2c7fe813b84f75f006041d8f20: Status 404 returned error can't find the container with id 01f98e234ff893c3bebf86f660bdb39ec9a79f2c7fe813b84f75f006041d8f20
Jul 28 18:33:51.329799 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:51.329756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" event={"ID":"fd44e575-fc79-4f46-bc78-00028b89f4f8","Type":"ContainerStarted","Data":"2e59d7a19322d56141d429fd689e323ad0ddbff73ac88e97aee0d35833d2bfcf"}
Jul 28 18:33:51.330223 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:51.329819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" event={"ID":"fd44e575-fc79-4f46-bc78-00028b89f4f8","Type":"ContainerStarted","Data":"01f98e234ff893c3bebf86f660bdb39ec9a79f2c7fe813b84f75f006041d8f20"}
Jul 28 18:33:55.353932 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:55.353896    2579 generic.go:356] "Generic (PLEG): container finished" podID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerID="2e59d7a19322d56141d429fd689e323ad0ddbff73ac88e97aee0d35833d2bfcf" exitCode=0
Jul 28 18:33:55.354457 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:55.353976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" event={"ID":"fd44e575-fc79-4f46-bc78-00028b89f4f8","Type":"ContainerDied","Data":"2e59d7a19322d56141d429fd689e323ad0ddbff73ac88e97aee0d35833d2bfcf"}
Jul 28 18:33:55.677782 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:55.677687    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"]
Jul 28 18:33:56.362324 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:56.362290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" event={"ID":"fd44e575-fc79-4f46-bc78-00028b89f4f8","Type":"ContainerStarted","Data":"08c2445b8eecb7593c508b622d3b67a1a70a933eeeb67305ec02f45283905632"}
Jul 28 18:33:56.385768 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:56.385665    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podStartSLOduration=2.7017193390000003 podStartE2EDuration="7.385646877s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.683927538s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:33:56.383663963 +0000 UTC m=+5024.612863058" watchObservedRunningTime="2026-07-28 18:33:56.385646877 +0000 UTC m=+5024.614845972"
Jul 28 18:33:56.866103 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:56.866055    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj" podUID="2095e6c0-9ff5-4332-a02d-7eb973625488" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:33:57.214979 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:33:57.214894    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:34:00.214618 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:00.214580    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:34:00.216180 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:00.216146    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:34:06.866264 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:06.866219    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj" podUID="2095e6c0-9ff5-4332-a02d-7eb973625488" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:34:10.215474 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:10.215431    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:34:10.697473 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:10.697424    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj" podUID="2095e6c0-9ff5-4332-a02d-7eb973625488" containerName="main" containerID="cri-o://2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30" gracePeriod=45
Jul 28 18:34:10.959350 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:10.959273    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:34:11.039924 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.039896    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2095e6c0-9ff5-4332-a02d-7eb973625488-tls-certs\") pod \"2095e6c0-9ff5-4332-a02d-7eb973625488\" (UID: \"2095e6c0-9ff5-4332-a02d-7eb973625488\") "
Jul 28 18:34:11.039924 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.039929    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kd7x9\" (UniqueName: \"kubernetes.io/projected/2095e6c0-9ff5-4332-a02d-7eb973625488-kube-api-access-kd7x9\") pod \"2095e6c0-9ff5-4332-a02d-7eb973625488\" (UID: \"2095e6c0-9ff5-4332-a02d-7eb973625488\") "
Jul 28 18:34:11.042138 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.042100    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2095e6c0-9ff5-4332-a02d-7eb973625488-kube-api-access-kd7x9" (OuterVolumeSpecName: "kube-api-access-kd7x9") pod "2095e6c0-9ff5-4332-a02d-7eb973625488" (UID: "2095e6c0-9ff5-4332-a02d-7eb973625488"). InnerVolumeSpecName "kube-api-access-kd7x9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:34:11.042255 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.042182    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2095e6c0-9ff5-4332-a02d-7eb973625488-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2095e6c0-9ff5-4332-a02d-7eb973625488" (UID: "2095e6c0-9ff5-4332-a02d-7eb973625488"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:34:11.141259 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.141222    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2095e6c0-9ff5-4332-a02d-7eb973625488-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:34:11.141259 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.141254    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd7x9\" (UniqueName: \"kubernetes.io/projected/2095e6c0-9ff5-4332-a02d-7eb973625488-kube-api-access-kd7x9\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:34:11.440763 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.440729    2579 generic.go:356] "Generic (PLEG): container finished" podID="2095e6c0-9ff5-4332-a02d-7eb973625488" containerID="2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30" exitCode=0
Jul 28 18:34:11.441203 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.440778    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"
Jul 28 18:34:11.441203 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.440803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj" event={"ID":"2095e6c0-9ff5-4332-a02d-7eb973625488","Type":"ContainerDied","Data":"2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30"}
Jul 28 18:34:11.441203 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.440846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj" event={"ID":"2095e6c0-9ff5-4332-a02d-7eb973625488","Type":"ContainerDied","Data":"6f941384108bde674cbcda33c0309b41bcb316d3790e0c66c0984908c2178974"}
Jul 28 18:34:11.441203 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.440870    2579 scope.go:117] "RemoveContainer" containerID="2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30"
Jul 28 18:34:11.450612 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.450589    2579 scope.go:117] "RemoveContainer" containerID="2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30"
Jul 28 18:34:11.450874 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:34:11.450852    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30\": container with ID starting with 2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30 not found: ID does not exist" containerID="2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30"
Jul 28 18:34:11.450944 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.450882    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30"} err="failed to get container status \"2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30\": rpc error: code = NotFound desc = could not find container \"2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30\": container with ID starting with 2cb5fe6e8ec59f96798e0b83b0ba3b82b56a974e94b8a6d190078d2209f92e30 not found: ID does not exist"
Jul 28 18:34:11.462715 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.462688    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"]
Jul 28 18:34:11.466751 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:11.466731    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-router-scheduler-5b6578b469-wxfkj"]
Jul 28 18:34:12.327847 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:12.327813    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2095e6c0-9ff5-4332-a02d-7eb973625488" path="/var/lib/kubelet/pods/2095e6c0-9ff5-4332-a02d-7eb973625488/volumes"
Jul 28 18:34:20.215557 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:20.215508    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:34:24.718280 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.718247    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"]
Jul 28 18:34:24.719553 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.719520    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2095e6c0-9ff5-4332-a02d-7eb973625488" containerName="main"
Jul 28 18:34:24.719553 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.719550    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2095e6c0-9ff5-4332-a02d-7eb973625488" containerName="main"
Jul 28 18:34:24.719714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.719661    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2095e6c0-9ff5-4332-a02d-7eb973625488" containerName="main"
Jul 28 18:34:24.723560 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.723537    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:34:24.726340 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.726312    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 28 18:34:24.727614 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.727589    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-472qz\""
Jul 28 18:34:24.727731 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.727632    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 28 18:34:24.727731 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.727644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-epp-sa-dockercfg-pmjj6\""
Jul 28 18:34:24.727908 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.727891    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 28 18:34:24.732470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.732435    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"]
Jul 28 18:34:24.769379 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.769353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452ec6b4-e954-428f-b5de-977d0f02ecb3-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c\" (UID: \"452ec6b4-e954-428f-b5de-977d0f02ecb3\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:34:24.769511 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.769478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rxjp\" (UniqueName: \"kubernetes.io/projected/452ec6b4-e954-428f-b5de-977d0f02ecb3-kube-api-access-9rxjp\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c\" (UID: \"452ec6b4-e954-428f-b5de-977d0f02ecb3\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:34:24.870324 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.870288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rxjp\" (UniqueName: \"kubernetes.io/projected/452ec6b4-e954-428f-b5de-977d0f02ecb3-kube-api-access-9rxjp\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c\" (UID: \"452ec6b4-e954-428f-b5de-977d0f02ecb3\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:34:24.870502 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.870361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452ec6b4-e954-428f-b5de-977d0f02ecb3-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c\" (UID: \"452ec6b4-e954-428f-b5de-977d0f02ecb3\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:34:24.873046 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.873023    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452ec6b4-e954-428f-b5de-977d0f02ecb3-tls-certs\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c\" (UID: \"452ec6b4-e954-428f-b5de-977d0f02ecb3\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:34:24.879410 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:24.879384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rxjp\" (UniqueName: \"kubernetes.io/projected/452ec6b4-e954-428f-b5de-977d0f02ecb3-kube-api-access-9rxjp\") pod \"autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c\" (UID: \"452ec6b4-e954-428f-b5de-977d0f02ecb3\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:34:25.038901 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:25.038814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:34:25.192364 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:25.191359    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"]
Jul 28 18:34:25.205597 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:34:25.205574    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452ec6b4_e954_428f_b5de_977d0f02ecb3.slice/crio-d79dca5ae6ace8fc217005cb89d787fc07a0787956d3d783c2a45c675e1611f5 WatchSource:0}: Error finding container d79dca5ae6ace8fc217005cb89d787fc07a0787956d3d783c2a45c675e1611f5: Status 404 returned error can't find the container with id d79dca5ae6ace8fc217005cb89d787fc07a0787956d3d783c2a45c675e1611f5
Jul 28 18:34:25.519562 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:25.519514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c" event={"ID":"452ec6b4-e954-428f-b5de-977d0f02ecb3","Type":"ContainerStarted","Data":"50147f1b7ae3d8305617a449991b496d0c29d2aeaa659b3082786daf7f67322a"}
Jul 28 18:34:25.519772 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:25.519603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c" event={"ID":"452ec6b4-e954-428f-b5de-977d0f02ecb3","Type":"ContainerStarted","Data":"d79dca5ae6ace8fc217005cb89d787fc07a0787956d3d783c2a45c675e1611f5"}
Jul 28 18:34:25.519772 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:25.519624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:34:25.537837 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:25.537782    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c" podStartSLOduration=1.537768461 podStartE2EDuration="1.537768461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:34:25.536379633 +0000 UTC m=+5053.765578731" watchObservedRunningTime="2026-07-28 18:34:25.537768461 +0000 UTC m=+5053.766967557"
Jul 28 18:34:30.215770 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:30.215729    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:34:40.214886 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:40.214828    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:34:50.215584 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:50.215538    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:34:56.527901 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:34:56.527864    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:35:00.215531 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:35:00.215477    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:35:10.215751 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:35:10.215710    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:35:12.690574 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:35:12.690536    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:35:12.720801 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:35:12.720779    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:35:20.215714 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:35:20.215662    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:35:30.215466 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:35:30.215409    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:35:40.215800 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:35:40.215704    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": dial tcp 10.132.0.98:8000: connect: connection refused"
Jul 28 18:35:50.225468 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:35:50.225434    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:35:50.233648 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:35:50.233624    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:35:54.413779 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:35:54.413740    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"]
Jul 28 18:36:09.437180 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:09.437126    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main" containerID="cri-o://08c2445b8eecb7593c508b622d3b67a1a70a933eeeb67305ec02f45283905632" gracePeriod=45
Jul 28 18:36:10.065925 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.065893    2579 generic.go:356] "Generic (PLEG): container finished" podID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerID="08c2445b8eecb7593c508b622d3b67a1a70a933eeeb67305ec02f45283905632" exitCode=0
Jul 28 18:36:10.066138 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.065963    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" event={"ID":"fd44e575-fc79-4f46-bc78-00028b89f4f8","Type":"ContainerDied","Data":"08c2445b8eecb7593c508b622d3b67a1a70a933eeeb67305ec02f45283905632"}
Jul 28 18:36:10.188294 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.188269    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:36:10.351709 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.351678    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrkxr\" (UniqueName: \"kubernetes.io/projected/fd44e575-fc79-4f46-bc78-00028b89f4f8-kube-api-access-hrkxr\") pod \"fd44e575-fc79-4f46-bc78-00028b89f4f8\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") "
Jul 28 18:36:10.351878 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.351742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd44e575-fc79-4f46-bc78-00028b89f4f8-tls-certs\") pod \"fd44e575-fc79-4f46-bc78-00028b89f4f8\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") "
Jul 28 18:36:10.351878 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.351780    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-model-cache\") pod \"fd44e575-fc79-4f46-bc78-00028b89f4f8\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") "
Jul 28 18:36:10.351878 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.351811    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-tmp-dir\") pod \"fd44e575-fc79-4f46-bc78-00028b89f4f8\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") "
Jul 28 18:36:10.351878 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.351828    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-home\") pod \"fd44e575-fc79-4f46-bc78-00028b89f4f8\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") "
Jul 28 18:36:10.351878 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.351849    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-dshm\") pod \"fd44e575-fc79-4f46-bc78-00028b89f4f8\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") "
Jul 28 18:36:10.352189 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.351889    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-kserve-provision-location\") pod \"fd44e575-fc79-4f46-bc78-00028b89f4f8\" (UID: \"fd44e575-fc79-4f46-bc78-00028b89f4f8\") "
Jul 28 18:36:10.352189 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.352136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd44e575-fc79-4f46-bc78-00028b89f4f8" (UID: "fd44e575-fc79-4f46-bc78-00028b89f4f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:36:10.352770 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.352740    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-home" (OuterVolumeSpecName: "home") pod "fd44e575-fc79-4f46-bc78-00028b89f4f8" (UID: "fd44e575-fc79-4f46-bc78-00028b89f4f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:36:10.354237 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.354213    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-dshm" (OuterVolumeSpecName: "dshm") pod "fd44e575-fc79-4f46-bc78-00028b89f4f8" (UID: "fd44e575-fc79-4f46-bc78-00028b89f4f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:36:10.354385 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.354257    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd44e575-fc79-4f46-bc78-00028b89f4f8-kube-api-access-hrkxr" (OuterVolumeSpecName: "kube-api-access-hrkxr") pod "fd44e575-fc79-4f46-bc78-00028b89f4f8" (UID: "fd44e575-fc79-4f46-bc78-00028b89f4f8"). InnerVolumeSpecName "kube-api-access-hrkxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:36:10.354385 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.354257    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd44e575-fc79-4f46-bc78-00028b89f4f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd44e575-fc79-4f46-bc78-00028b89f4f8" (UID: "fd44e575-fc79-4f46-bc78-00028b89f4f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:36:10.371138 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.371110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd44e575-fc79-4f46-bc78-00028b89f4f8" (UID: "fd44e575-fc79-4f46-bc78-00028b89f4f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:36:10.419310 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.419280    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd44e575-fc79-4f46-bc78-00028b89f4f8" (UID: "fd44e575-fc79-4f46-bc78-00028b89f4f8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:36:10.453856 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.453829    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrkxr\" (UniqueName: \"kubernetes.io/projected/fd44e575-fc79-4f46-bc78-00028b89f4f8-kube-api-access-hrkxr\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:36:10.454250 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.453972    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd44e575-fc79-4f46-bc78-00028b89f4f8-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:36:10.454250 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.453992    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:36:10.454250 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.454006    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:36:10.454250 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.454018    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:36:10.454250 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.454029    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:36:10.454250 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:10.454042    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd44e575-fc79-4f46-bc78-00028b89f4f8-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:36:11.072015 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:11.071978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b" event={"ID":"fd44e575-fc79-4f46-bc78-00028b89f4f8","Type":"ContainerDied","Data":"01f98e234ff893c3bebf86f660bdb39ec9a79f2c7fe813b84f75f006041d8f20"}
Jul 28 18:36:11.072255 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:11.072027    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"
Jul 28 18:36:11.072255 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:11.072042    2579 scope.go:117] "RemoveContainer" containerID="08c2445b8eecb7593c508b622d3b67a1a70a933eeeb67305ec02f45283905632"
Jul 28 18:36:11.081260 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:11.081238    2579 scope.go:117] "RemoveContainer" containerID="2e59d7a19322d56141d429fd689e323ad0ddbff73ac88e97aee0d35833d2bfcf"
Jul 28 18:36:11.095809 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:11.095786    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"]
Jul 28 18:36:11.099419 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:11.099398    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-6c4d7c779dk557b"]
Jul 28 18:36:12.320650 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:12.320616    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" path="/var/lib/kubelet/pods/fd44e575-fc79-4f46-bc78-00028b89f4f8/volumes"
Jul 28 18:36:23.206603 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.206567    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"]
Jul 28 18:36:23.207008 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.206994    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="storage-initializer"
Jul 28 18:36:23.207054 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.207010    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="storage-initializer"
Jul 28 18:36:23.207054 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.207018    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main"
Jul 28 18:36:23.207054 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.207025    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main"
Jul 28 18:36:23.207198 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.207119    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd44e575-fc79-4f46-bc78-00028b89f4f8" containerName="main"
Jul 28 18:36:23.211210 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.211190    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.213963 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.213941    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Jul 28 18:36:23.213963 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.213957    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-cp56k\""
Jul 28 18:36:23.214183 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.214005    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Jul 28 18:36:23.214274 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.214257    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 28 18:36:23.214326 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.214304    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"seaweedfs-s3-creds\""
Jul 28 18:36:23.224153 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.224132    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"]
Jul 28 18:36:23.265051 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.265016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-dshm\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.265251 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.265113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-tmp-dir\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.265251 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.265159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.265251 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.265216    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcvv7\" (UniqueName: \"kubernetes.io/projected/edf6b858-ef85-4a37-8742-7348d88b89ec-kube-api-access-vcvv7\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.265394 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.265287    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-home\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.265394 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.265307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-model-cache\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.265394 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.265332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf6b858-ef85-4a37-8742-7348d88b89ec-tls-certs\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.366608 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.366568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf6b858-ef85-4a37-8742-7348d88b89ec-tls-certs\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.366807 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.366639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-dshm\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.366807 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.366756    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-tmp-dir\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.366807 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.366797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.366998 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.366841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcvv7\" (UniqueName: \"kubernetes.io/projected/edf6b858-ef85-4a37-8742-7348d88b89ec-kube-api-access-vcvv7\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.366998 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.366915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-home\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.366998 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.366935    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-model-cache\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.367260 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.367229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-tmp-dir\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.367260 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.367239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.367399 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.367285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-model-cache\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.367399 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.367372    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-home\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.369015 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.368997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-dshm\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.369321 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.369306    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf6b858-ef85-4a37-8742-7348d88b89ec-tls-certs\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.375248 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.375230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcvv7\" (UniqueName: \"kubernetes.io/projected/edf6b858-ef85-4a37-8742-7348d88b89ec-kube-api-access-vcvv7\") pod \"custom-route-timeout-test-kserve-547f866547-hswxt\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.522221 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.522133    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:23.663010 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:23.662987    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"]
Jul 28 18:36:23.665495 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:36:23.665449    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf6b858_ef85_4a37_8742_7348d88b89ec.slice/crio-a9fc793b18c9db282bc3be59cb6dc4983e47be2fcd58101a63bf8e1fca787d7a WatchSource:0}: Error finding container a9fc793b18c9db282bc3be59cb6dc4983e47be2fcd58101a63bf8e1fca787d7a: Status 404 returned error can't find the container with id a9fc793b18c9db282bc3be59cb6dc4983e47be2fcd58101a63bf8e1fca787d7a
Jul 28 18:36:24.134583 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:24.134540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" event={"ID":"edf6b858-ef85-4a37-8742-7348d88b89ec","Type":"ContainerStarted","Data":"2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1"}
Jul 28 18:36:24.134811 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:24.134606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" event={"ID":"edf6b858-ef85-4a37-8742-7348d88b89ec","Type":"ContainerStarted","Data":"a9fc793b18c9db282bc3be59cb6dc4983e47be2fcd58101a63bf8e1fca787d7a"}
Jul 28 18:36:30.167820 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:30.167788    2579 generic.go:356] "Generic (PLEG): container finished" podID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerID="2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1" exitCode=0
Jul 28 18:36:30.168228 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:30.167855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" event={"ID":"edf6b858-ef85-4a37-8742-7348d88b89ec","Type":"ContainerDied","Data":"2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1"}
Jul 28 18:36:31.174561 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:31.174521    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" event={"ID":"edf6b858-ef85-4a37-8742-7348d88b89ec","Type":"ContainerStarted","Data":"cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37"}
Jul 28 18:36:31.197476 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:31.197422    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podStartSLOduration=2.424129529 podStartE2EDuration="8.19740562s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.773276091s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:36:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:36:31.194931933 +0000 UTC m=+5179.424131028" watchObservedRunningTime="2026-07-28 18:36:31.19740562 +0000 UTC m=+5179.426604716"
Jul 28 18:36:31.522936 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:31.522846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:33.523072 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:33.523038    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:36:33.524371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:33.524344    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:36:43.522583 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:43.522540    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:36:53.523000 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:36:53.522949    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:37:03.523218 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:37:03.523170    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:37:13.522992 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:37:13.522942    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:37:23.522554 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:37:23.522510    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:37:33.523068 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:37:33.523027    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:37:43.523266 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:37:43.523218    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:37:53.522925 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:37:53.522881    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:38:03.522761 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:03.522709    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:38:13.522532 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:13.522488    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": dial tcp 10.132.0.100:8000: connect: connection refused"
Jul 28 18:38:23.532501 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:23.532468    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:38:23.540239 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:23.540212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:38:27.827242 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:27.827209    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"]
Jul 28 18:38:42.850808 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:42.850715    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" containerID="cri-o://cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37" gracePeriod=45
Jul 28 18:38:43.607267 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.607213    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:38:43.687254 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.687220    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-kserve-provision-location\") pod \"edf6b858-ef85-4a37-8742-7348d88b89ec\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") "
Jul 28 18:38:43.687423 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.687311    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf6b858-ef85-4a37-8742-7348d88b89ec-tls-certs\") pod \"edf6b858-ef85-4a37-8742-7348d88b89ec\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") "
Jul 28 18:38:43.687423 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.687369    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-tmp-dir\") pod \"edf6b858-ef85-4a37-8742-7348d88b89ec\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") "
Jul 28 18:38:43.687423 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.687396    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcvv7\" (UniqueName: \"kubernetes.io/projected/edf6b858-ef85-4a37-8742-7348d88b89ec-kube-api-access-vcvv7\") pod \"edf6b858-ef85-4a37-8742-7348d88b89ec\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") "
Jul 28 18:38:43.687557 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.687428    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-dshm\") pod \"edf6b858-ef85-4a37-8742-7348d88b89ec\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") "
Jul 28 18:38:43.687557 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.687457    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-home\") pod \"edf6b858-ef85-4a37-8742-7348d88b89ec\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") "
Jul 28 18:38:43.687557 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.687494    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-model-cache\") pod \"edf6b858-ef85-4a37-8742-7348d88b89ec\" (UID: \"edf6b858-ef85-4a37-8742-7348d88b89ec\") "
Jul 28 18:38:43.688034 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.687984    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-model-cache" (OuterVolumeSpecName: "model-cache") pod "edf6b858-ef85-4a37-8742-7348d88b89ec" (UID: "edf6b858-ef85-4a37-8742-7348d88b89ec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:38:43.688313 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.688292    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-home" (OuterVolumeSpecName: "home") pod "edf6b858-ef85-4a37-8742-7348d88b89ec" (UID: "edf6b858-ef85-4a37-8742-7348d88b89ec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:38:43.690288 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.690250    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf6b858-ef85-4a37-8742-7348d88b89ec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "edf6b858-ef85-4a37-8742-7348d88b89ec" (UID: "edf6b858-ef85-4a37-8742-7348d88b89ec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:38:43.690402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.690316    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-dshm" (OuterVolumeSpecName: "dshm") pod "edf6b858-ef85-4a37-8742-7348d88b89ec" (UID: "edf6b858-ef85-4a37-8742-7348d88b89ec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:38:43.690402 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.690346    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf6b858-ef85-4a37-8742-7348d88b89ec-kube-api-access-vcvv7" (OuterVolumeSpecName: "kube-api-access-vcvv7") pod "edf6b858-ef85-4a37-8742-7348d88b89ec" (UID: "edf6b858-ef85-4a37-8742-7348d88b89ec"). InnerVolumeSpecName "kube-api-access-vcvv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:38:43.700857 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.700833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "edf6b858-ef85-4a37-8742-7348d88b89ec" (UID: "edf6b858-ef85-4a37-8742-7348d88b89ec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:38:43.751336 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.751290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "edf6b858-ef85-4a37-8742-7348d88b89ec" (UID: "edf6b858-ef85-4a37-8742-7348d88b89ec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:38:43.788451 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.788424    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:38:43.788451 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.788447    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/edf6b858-ef85-4a37-8742-7348d88b89ec-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:38:43.788616 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.788467    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:38:43.788616 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.788476    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcvv7\" (UniqueName: \"kubernetes.io/projected/edf6b858-ef85-4a37-8742-7348d88b89ec-kube-api-access-vcvv7\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:38:43.788616 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.788484    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:38:43.788616 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.788492    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:38:43.788616 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.788499    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/edf6b858-ef85-4a37-8742-7348d88b89ec-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:38:43.895682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.895595    2579 generic.go:356] "Generic (PLEG): container finished" podID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerID="cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37" exitCode=0
Jul 28 18:38:43.895682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.895650    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"
Jul 28 18:38:43.895682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.895658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" event={"ID":"edf6b858-ef85-4a37-8742-7348d88b89ec","Type":"ContainerDied","Data":"cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37"}
Jul 28 18:38:43.896292 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.895734    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" event={"ID":"edf6b858-ef85-4a37-8742-7348d88b89ec","Type":"ContainerDied","Data":"a9fc793b18c9db282bc3be59cb6dc4983e47be2fcd58101a63bf8e1fca787d7a"}
Jul 28 18:38:43.896292 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.895759    2579 scope.go:117] "RemoveContainer" containerID="cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37"
Jul 28 18:38:43.905393 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.905375    2579 scope.go:117] "RemoveContainer" containerID="2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1"
Jul 28 18:38:43.913328 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.913309    2579 scope.go:117] "RemoveContainer" containerID="cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37"
Jul 28 18:38:43.913566 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:38:43.913549    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37\": container with ID starting with cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37 not found: ID does not exist" containerID="cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37"
Jul 28 18:38:43.913607 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.913573    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37"} err="failed to get container status \"cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37\": rpc error: code = NotFound desc = could not find container \"cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37\": container with ID starting with cf3cd78214759a6147493eb02401f60c4bf647df8a045cda7cb0967b6d4d9b37 not found: ID does not exist"
Jul 28 18:38:43.913607 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.913588    2579 scope.go:117] "RemoveContainer" containerID="2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1"
Jul 28 18:38:43.913807 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:38:43.913791    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1\": container with ID starting with 2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1 not found: ID does not exist" containerID="2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1"
Jul 28 18:38:43.913853 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.913812    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1"} err="failed to get container status \"2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1\": rpc error: code = NotFound desc = could not find container \"2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1\": container with ID starting with 2785c0d1a5d1b828fd44eb0c47ef49d2a48ab6159ee692d74ca4a76f2e3ab4b1 not found: ID does not exist"
Jul 28 18:38:43.919706 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.919684    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"]
Jul 28 18:38:43.923148 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:43.923123    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt"]
Jul 28 18:38:44.320702 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:44.320663    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" path="/var/lib/kubelet/pods/edf6b858-ef85-4a37-8742-7348d88b89ec/volumes"
Jul 28 18:38:44.533732 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:38:44.533693    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-547f866547-hswxt" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 18:39:12.820283 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.820241    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"]
Jul 28 18:39:12.820839 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.820819    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="storage-initializer"
Jul 28 18:39:12.820919 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.820844    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="storage-initializer"
Jul 28 18:39:12.820919 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.820874    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main"
Jul 28 18:39:12.820919 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.820884    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main"
Jul 28 18:39:12.821096 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.820973    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf6b858-ef85-4a37-8742-7348d88b89ec" containerName="main"
Jul 28 18:39:12.823406 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.823378    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:12.826461 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.826270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Jul 28 18:39:12.826461 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.826377    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 28 18:39:12.826461 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.826395    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Jul 28 18:39:12.826461 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.826403    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"seaweedfs-s3-creds\""
Jul 28 18:39:12.826461 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.826413    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-sbvs9\""
Jul 28 18:39:12.837014 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.836991    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"]
Jul 28 18:39:12.932190 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.932157    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-model-cache\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:12.932410 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.932220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-dshm\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:12.932410 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.932273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-home\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:12.932410 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.932295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/241c0c0a-5593-4842-b2d6-31a425ea06d8-tls-certs\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:12.932410 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.932362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5tb\" (UniqueName: \"kubernetes.io/projected/241c0c0a-5593-4842-b2d6-31a425ea06d8-kube-api-access-9r5tb\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:12.932410 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.932402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-kserve-provision-location\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:12.932632 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:12.932461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-tmp-dir\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.033503 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.033464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-kserve-provision-location\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.033723 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.033529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-tmp-dir\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.033723 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.033572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-model-cache\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.033723 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.033615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-dshm\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.033723 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.033663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-home\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.033723 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.033687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/241c0c0a-5593-4842-b2d6-31a425ea06d8-tls-certs\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.033961 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.033727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5tb\" (UniqueName: \"kubernetes.io/projected/241c0c0a-5593-4842-b2d6-31a425ea06d8-kube-api-access-9r5tb\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.033961 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.033880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-kserve-provision-location\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.033961 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.033931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-tmp-dir\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.034254 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.034228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-model-cache\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.034254 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.034246    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-home\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.036063 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.036041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-dshm\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.036449 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.036429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/241c0c0a-5593-4842-b2d6-31a425ea06d8-tls-certs\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.042663 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.042641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5tb\" (UniqueName: \"kubernetes.io/projected/241c0c0a-5593-4842-b2d6-31a425ea06d8-kube-api-access-9r5tb\") pod \"router-with-refs-test-kserve-5fcc78bcc5-c6vrv\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.135003 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.134910    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:13.276973 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:13.276943    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"]
Jul 28 18:39:13.278961 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:39:13.278937    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241c0c0a_5593_4842_b2d6_31a425ea06d8.slice/crio-449229a8b8225a28b1e6b4acd425c5e83b1ed8e9f481bbbdd3d06f5b1b00ce9a WatchSource:0}: Error finding container 449229a8b8225a28b1e6b4acd425c5e83b1ed8e9f481bbbdd3d06f5b1b00ce9a: Status 404 returned error can't find the container with id 449229a8b8225a28b1e6b4acd425c5e83b1ed8e9f481bbbdd3d06f5b1b00ce9a
Jul 28 18:39:14.046020 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:14.045982    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" event={"ID":"241c0c0a-5593-4842-b2d6-31a425ea06d8","Type":"ContainerStarted","Data":"6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4"}
Jul 28 18:39:14.046501 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:14.046050    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" event={"ID":"241c0c0a-5593-4842-b2d6-31a425ea06d8","Type":"ContainerStarted","Data":"449229a8b8225a28b1e6b4acd425c5e83b1ed8e9f481bbbdd3d06f5b1b00ce9a"}
Jul 28 18:39:23.092318 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:23.092281    2579 generic.go:356] "Generic (PLEG): container finished" podID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerID="6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4" exitCode=0
Jul 28 18:39:23.092700 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:23.092319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" event={"ID":"241c0c0a-5593-4842-b2d6-31a425ea06d8","Type":"ContainerDied","Data":"6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4"}
Jul 28 18:39:24.101401 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:24.101358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" event={"ID":"241c0c0a-5593-4842-b2d6-31a425ea06d8","Type":"ContainerStarted","Data":"6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8"}
Jul 28 18:39:24.132761 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:24.132710    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podStartSLOduration=2.775376504 podStartE2EDuration="12.132695233s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.357318729s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:39:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:39:24.129346765 +0000 UTC m=+5352.358545884" watchObservedRunningTime="2026-07-28 18:39:24.132695233 +0000 UTC m=+5352.361894329"
Jul 28 18:39:24.135565 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:24.135538    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:33.135283 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:33.135244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:39:33.136931 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:33.136901    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:39:43.135963 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:43.135913    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:39:53.136121 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:39:53.136051    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:40:03.136057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:40:03.136004    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:40:12.724039 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:40:12.724008    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:40:12.756506 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:40:12.756476    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:40:13.136298 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:40:13.136259    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:40:23.135536 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:40:23.135487    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:40:33.135994 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:40:33.135953    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:40:43.135893 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:40:43.135842    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:40:53.136207 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:40:53.136166    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:41:03.135587 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:41:03.135547    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:41:13.135532 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:41:13.135489    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:41:23.145811 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:41:23.145775    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:41:23.153828 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:41:23.153803    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:45:12.755880 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:12.755808    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:45:12.789332 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:12.789308    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:45:31.833527 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:31.833492    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"]
Jul 28 18:45:46.856939 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:46.856896    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" containerID="cri-o://6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8" gracePeriod=45
Jul 28 18:45:47.147228 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.147136    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/health\": dial tcp 10.132.0.101:8000: connect: connection refused"
Jul 28 18:45:47.634978 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.634951    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:45:47.673372 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.673344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-home\") pod \"241c0c0a-5593-4842-b2d6-31a425ea06d8\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") "
Jul 28 18:45:47.673543 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.673378    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/241c0c0a-5593-4842-b2d6-31a425ea06d8-tls-certs\") pod \"241c0c0a-5593-4842-b2d6-31a425ea06d8\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") "
Jul 28 18:45:47.673543 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.673401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-model-cache\") pod \"241c0c0a-5593-4842-b2d6-31a425ea06d8\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") "
Jul 28 18:45:47.673543 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.673435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-tmp-dir\") pod \"241c0c0a-5593-4842-b2d6-31a425ea06d8\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") "
Jul 28 18:45:47.673543 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.673476    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-kserve-provision-location\") pod \"241c0c0a-5593-4842-b2d6-31a425ea06d8\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") "
Jul 28 18:45:47.673543 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.673509    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-dshm\") pod \"241c0c0a-5593-4842-b2d6-31a425ea06d8\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") "
Jul 28 18:45:47.673820 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.673588    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r5tb\" (UniqueName: \"kubernetes.io/projected/241c0c0a-5593-4842-b2d6-31a425ea06d8-kube-api-access-9r5tb\") pod \"241c0c0a-5593-4842-b2d6-31a425ea06d8\" (UID: \"241c0c0a-5593-4842-b2d6-31a425ea06d8\") "
Jul 28 18:45:47.674019 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.673973    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "241c0c0a-5593-4842-b2d6-31a425ea06d8" (UID: "241c0c0a-5593-4842-b2d6-31a425ea06d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:47.674166 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.674095    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-home" (OuterVolumeSpecName: "home") pod "241c0c0a-5593-4842-b2d6-31a425ea06d8" (UID: "241c0c0a-5593-4842-b2d6-31a425ea06d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:47.675821 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.675800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/241c0c0a-5593-4842-b2d6-31a425ea06d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "241c0c0a-5593-4842-b2d6-31a425ea06d8" (UID: "241c0c0a-5593-4842-b2d6-31a425ea06d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:45:47.675918 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.675845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241c0c0a-5593-4842-b2d6-31a425ea06d8-kube-api-access-9r5tb" (OuterVolumeSpecName: "kube-api-access-9r5tb") pod "241c0c0a-5593-4842-b2d6-31a425ea06d8" (UID: "241c0c0a-5593-4842-b2d6-31a425ea06d8"). InnerVolumeSpecName "kube-api-access-9r5tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:45:47.676974 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.676946    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-dshm" (OuterVolumeSpecName: "dshm") pod "241c0c0a-5593-4842-b2d6-31a425ea06d8" (UID: "241c0c0a-5593-4842-b2d6-31a425ea06d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:47.687212 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.687187    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "241c0c0a-5593-4842-b2d6-31a425ea06d8" (UID: "241c0c0a-5593-4842-b2d6-31a425ea06d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:47.748778 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.748697    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "241c0c0a-5593-4842-b2d6-31a425ea06d8" (UID: "241c0c0a-5593-4842-b2d6-31a425ea06d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:45:47.774671 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.774640    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:45:47.774671 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.774669    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r5tb\" (UniqueName: \"kubernetes.io/projected/241c0c0a-5593-4842-b2d6-31a425ea06d8-kube-api-access-9r5tb\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:45:47.774839 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.774679    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:45:47.774839 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.774687    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/241c0c0a-5593-4842-b2d6-31a425ea06d8-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:45:47.774839 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.774713    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:45:47.774839 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.774721    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:45:47.774839 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:47.774729    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/241c0c0a-5593-4842-b2d6-31a425ea06d8-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:45:48.003600 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.003508    2579 generic.go:356] "Generic (PLEG): container finished" podID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerID="6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8" exitCode=0
Jul 28 18:45:48.003600 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.003573    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"
Jul 28 18:45:48.003600 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.003593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" event={"ID":"241c0c0a-5593-4842-b2d6-31a425ea06d8","Type":"ContainerDied","Data":"6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8"}
Jul 28 18:45:48.004284 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.003672    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv" event={"ID":"241c0c0a-5593-4842-b2d6-31a425ea06d8","Type":"ContainerDied","Data":"449229a8b8225a28b1e6b4acd425c5e83b1ed8e9f481bbbdd3d06f5b1b00ce9a"}
Jul 28 18:45:48.004284 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.003697    2579 scope.go:117] "RemoveContainer" containerID="6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8"
Jul 28 18:45:48.012981 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.012948    2579 scope.go:117] "RemoveContainer" containerID="6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4"
Jul 28 18:45:48.020934 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.020917    2579 scope.go:117] "RemoveContainer" containerID="6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8"
Jul 28 18:45:48.021195 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:45:48.021176    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8\": container with ID starting with 6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8 not found: ID does not exist" containerID="6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8"
Jul 28 18:45:48.021268 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.021202    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8"} err="failed to get container status \"6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8\": rpc error: code = NotFound desc = could not find container \"6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8\": container with ID starting with 6338fabaa67df3cb4b38fe36bb5b4c2b25f9d90143c9d27e3a3e3721b6d765e8 not found: ID does not exist"
Jul 28 18:45:48.021268 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.021220    2579 scope.go:117] "RemoveContainer" containerID="6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4"
Jul 28 18:45:48.021466 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:45:48.021448    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4\": container with ID starting with 6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4 not found: ID does not exist" containerID="6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4"
Jul 28 18:45:48.021506 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.021471    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4"} err="failed to get container status \"6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4\": rpc error: code = NotFound desc = could not find container \"6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4\": container with ID starting with 6e6626b09f29fb13be4ad7a4ee2a115fa338973f3c158abeb9b584a55d17c2d4 not found: ID does not exist"
Jul 28 18:45:48.025908 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.025888    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"]
Jul 28 18:45:48.029744 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.029725    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-5fcc78bcc5-c6vrv"]
Jul 28 18:45:48.320528 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:45:48.320497    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" path="/var/lib/kubelet/pods/241c0c0a-5593-4842-b2d6-31a425ea06d8/volumes"
Jul 28 18:46:08.762391 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.762315    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"]
Jul 28 18:46:08.762757 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.762731    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main"
Jul 28 18:46:08.762757 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.762745    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main"
Jul 28 18:46:08.762757 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.762756    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="storage-initializer"
Jul 28 18:46:08.762868 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.762761    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="storage-initializer"
Jul 28 18:46:08.762868 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.762826    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="241c0c0a-5593-4842-b2d6-31a425ea06d8" containerName="main"
Jul 28 18:46:08.768299 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.768235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.770862 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.770835    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-cz6fn\""
Jul 28 18:46:08.771034 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.770892    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 28 18:46:08.771347 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.771328    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Jul 28 18:46:08.771931 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.771896    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"seaweedfs-s3-creds\""
Jul 28 18:46:08.772069 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.771940    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-hhpzw\""
Jul 28 18:46:08.772069 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.771965    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Jul 28 18:46:08.781476 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.781408    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"]
Jul 28 18:46:08.793575 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.793549    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"]
Jul 28 18:46:08.797397 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.797377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:08.807982 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.807958    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"]
Jul 28 18:46:08.869363 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.869328    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqk8\" (UniqueName: \"kubernetes.io/projected/3a885212-29b1-4ba8-a024-cc0bd7c21106-kube-api-access-ggqk8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.869536 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.869381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.869536 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.869411    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.869536 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.869440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.869536 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.869521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.869677 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.869541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.869677 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.869556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a885212-29b1-4ba8-a024-cc0bd7c21106-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970094 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:08.970295 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:08.970295 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqk8\" (UniqueName: \"kubernetes.io/projected/3a885212-29b1-4ba8-a024-cc0bd7c21106-kube-api-access-ggqk8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970295 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:08.970295 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970244    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:08.970295 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970572 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970572 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:08.970572 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970572 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2hsr\" (UniqueName: \"kubernetes.io/projected/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kube-api-access-t2hsr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:08.970572 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:08.970572 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970539    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970890 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970890 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a885212-29b1-4ba8-a024-cc0bd7c21106-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970890 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970890 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970759    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970890 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.970890 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.970887    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.972675 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.972654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.973031 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.973013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a885212-29b1-4ba8-a024-cc0bd7c21106-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:08.978111 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:08.978072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqk8\" (UniqueName: \"kubernetes.io/projected/3a885212-29b1-4ba8-a024-cc0bd7c21106-kube-api-access-ggqk8\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:09.071638 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.071600    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.071841 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.071647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.071841 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.071696    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.071841 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.071736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.071841 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.071782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.071841 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.071824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2hsr\" (UniqueName: \"kubernetes.io/projected/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kube-api-access-t2hsr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.072191 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.071857    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.072191 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.072029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.072191 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.072123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.072191 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.072177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.072501 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.072471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.074270 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.074250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.074410 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.074391    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.080118 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.080095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2hsr\" (UniqueName: \"kubernetes.io/projected/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kube-api-access-t2hsr\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.080314 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.080298    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:09.108049 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.108026    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:09.237217 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.237186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"]
Jul 28 18:46:09.239088 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:46:09.239052    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a885212_29b1_4ba8_a024_cc0bd7c21106.slice/crio-cc34c26bc924ddcb34334baee3d7307edc3ed74510f97c36b8c2f370ddc030a0 WatchSource:0}: Error finding container cc34c26bc924ddcb34334baee3d7307edc3ed74510f97c36b8c2f370ddc030a0: Status 404 returned error can't find the container with id cc34c26bc924ddcb34334baee3d7307edc3ed74510f97c36b8c2f370ddc030a0
Jul 28 18:46:09.252855 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:09.252834    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"]
Jul 28 18:46:09.255298 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:46:09.255262    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f8a118_dc0f_4959_bf50_1ee9a1d07f2d.slice/crio-cfd4cf7147e14a6310f08b008d0524a119d621ff6f07f78ed7c01db5a0cbe11d WatchSource:0}: Error finding container cfd4cf7147e14a6310f08b008d0524a119d621ff6f07f78ed7c01db5a0cbe11d: Status 404 returned error can't find the container with id cfd4cf7147e14a6310f08b008d0524a119d621ff6f07f78ed7c01db5a0cbe11d
Jul 28 18:46:10.125435 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:10.125391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" event={"ID":"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d","Type":"ContainerStarted","Data":"eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc"}
Jul 28 18:46:10.125951 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:10.125460    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" event={"ID":"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d","Type":"ContainerStarted","Data":"cfd4cf7147e14a6310f08b008d0524a119d621ff6f07f78ed7c01db5a0cbe11d"}
Jul 28 18:46:10.126886 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:10.126859    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" event={"ID":"3a885212-29b1-4ba8-a024-cc0bd7c21106","Type":"ContainerStarted","Data":"6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f"}
Jul 28 18:46:10.127011 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:10.126902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" event={"ID":"3a885212-29b1-4ba8-a024-cc0bd7c21106","Type":"ContainerStarted","Data":"cc34c26bc924ddcb34334baee3d7307edc3ed74510f97c36b8c2f370ddc030a0"}
Jul 28 18:46:10.127011 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:10.126939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:11.133388 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:11.133342    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" event={"ID":"3a885212-29b1-4ba8-a024-cc0bd7c21106","Type":"ContainerStarted","Data":"151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d"}
Jul 28 18:46:22.154187 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:22.154152    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:32.245729 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:32.245691    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerID="151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d" exitCode=0
Jul 28 18:46:32.246265 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:32.245784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" event={"ID":"3a885212-29b1-4ba8-a024-cc0bd7c21106","Type":"ContainerDied","Data":"151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d"}
Jul 28 18:46:33.254139 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:33.254097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" event={"ID":"3a885212-29b1-4ba8-a024-cc0bd7c21106","Type":"ContainerStarted","Data":"133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa"}
Jul 28 18:46:33.279152 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:33.279095    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podStartSLOduration=3.949427939 podStartE2EDuration="25.279060485s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.329632546s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:46:33.276005387 +0000 UTC m=+5781.505204484" watchObservedRunningTime="2026-07-28 18:46:33.279060485 +0000 UTC m=+5781.508259586"
Jul 28 18:46:38.283277 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:38.283244    2579 generic.go:356] "Generic (PLEG): container finished" podID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerID="eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc" exitCode=0
Jul 28 18:46:38.283682 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:38.283328    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" event={"ID":"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d","Type":"ContainerDied","Data":"eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc"}
Jul 28 18:46:39.080634 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:39.080597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:39.080943 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:39.080671    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:46:39.081392 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:39.081353    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:46:39.291183 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:39.291144    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" event={"ID":"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d","Type":"ContainerStarted","Data":"ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024"}
Jul 28 18:46:39.314522 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:39.314462    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podStartSLOduration=3.101829428 podStartE2EDuration="31.314442351s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.212612923s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:46:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:46:39.311398424 +0000 UTC m=+5787.540597520" watchObservedRunningTime="2026-07-28 18:46:39.314442351 +0000 UTC m=+5787.543641451"
Jul 28 18:46:49.081203 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:49.081155    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:46:49.108822 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:49.108784    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:49.108998 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:49.108857    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:46:49.110341 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:49.110313    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:46:59.080830 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:59.080780    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:46:59.109010 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:46:59.108961    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:47:09.081097 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:09.081027    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:47:09.109314 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:09.109267    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:47:19.081577 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:19.081529    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:47:19.109153 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:19.109101    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:47:29.081514 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:29.081463    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:47:29.109051 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:29.109011    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:47:39.081598 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:39.081487    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:47:39.108726 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:39.108680    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:47:49.081763 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:49.081709    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:47:49.108847 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:49.108808    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:47:59.080912 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:59.080853    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:47:59.108548 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:47:59.108509    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:48:09.081255 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:09.081212    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:48:09.109164 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:09.109130    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:48:19.081746 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:19.081696    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:48:19.108431 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:19.108381    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:48:29.081774 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:29.081718    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:48:29.109310 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:29.109265    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:48:39.081763 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:39.081702    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:48:39.108403 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:39.108349    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:48:49.080958 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:49.080902    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8001/health\": dial tcp 10.132.0.102:8001: connect: connection refused"
Jul 28 18:48:49.109476 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:49.109427    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:48:59.091247 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:59.091207    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:48:59.102811 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:59.102785    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:48:59.109003 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:48:59.108979    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.103:8000/health\": dial tcp 10.132.0.103:8000: connect: connection refused"
Jul 28 18:49:09.119219 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:09.119142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:49:09.127423 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:09.127380    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:49:16.033837 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:16.033795    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"]
Jul 28 18:49:16.527951 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:16.527910    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c" podUID="452ec6b4-e954-428f-b5de-977d0f02ecb3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:49:19.443369 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:19.443334    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"]
Jul 28 18:49:19.447237 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:19.447205    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"]
Jul 28 18:49:26.527307 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:26.527267    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c" podUID="452ec6b4-e954-428f-b5de-977d0f02ecb3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:49:31.056410 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:31.056344    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c" podUID="452ec6b4-e954-428f-b5de-977d0f02ecb3" containerName="main" containerID="cri-o://50147f1b7ae3d8305617a449991b496d0c29d2aeaa659b3082786daf7f67322a" gracePeriod=45
Jul 28 18:49:31.252667 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:31.252633    2579 generic.go:356] "Generic (PLEG): container finished" podID="452ec6b4-e954-428f-b5de-977d0f02ecb3" containerID="50147f1b7ae3d8305617a449991b496d0c29d2aeaa659b3082786daf7f67322a" exitCode=0
Jul 28 18:49:31.252813 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:31.252696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c" event={"ID":"452ec6b4-e954-428f-b5de-977d0f02ecb3","Type":"ContainerDied","Data":"50147f1b7ae3d8305617a449991b496d0c29d2aeaa659b3082786daf7f67322a"}
Jul 28 18:49:31.313264 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:31.313208    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:49:31.345005 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:31.344976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452ec6b4-e954-428f-b5de-977d0f02ecb3-tls-certs\") pod \"452ec6b4-e954-428f-b5de-977d0f02ecb3\" (UID: \"452ec6b4-e954-428f-b5de-977d0f02ecb3\") "
Jul 28 18:49:31.345180 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:31.345090    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rxjp\" (UniqueName: \"kubernetes.io/projected/452ec6b4-e954-428f-b5de-977d0f02ecb3-kube-api-access-9rxjp\") pod \"452ec6b4-e954-428f-b5de-977d0f02ecb3\" (UID: \"452ec6b4-e954-428f-b5de-977d0f02ecb3\") "
Jul 28 18:49:31.347305 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:31.347276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/452ec6b4-e954-428f-b5de-977d0f02ecb3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "452ec6b4-e954-428f-b5de-977d0f02ecb3" (UID: "452ec6b4-e954-428f-b5de-977d0f02ecb3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:49:31.347422 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:31.347298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452ec6b4-e954-428f-b5de-977d0f02ecb3-kube-api-access-9rxjp" (OuterVolumeSpecName: "kube-api-access-9rxjp") pod "452ec6b4-e954-428f-b5de-977d0f02ecb3" (UID: "452ec6b4-e954-428f-b5de-977d0f02ecb3"). InnerVolumeSpecName "kube-api-access-9rxjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:49:31.446573 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:31.446539    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/452ec6b4-e954-428f-b5de-977d0f02ecb3-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:31.446573 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:31.446571    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rxjp\" (UniqueName: \"kubernetes.io/projected/452ec6b4-e954-428f-b5de-977d0f02ecb3-kube-api-access-9rxjp\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:32.259987 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:32.259955    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"
Jul 28 18:49:32.260512 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:32.259983    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c" event={"ID":"452ec6b4-e954-428f-b5de-977d0f02ecb3","Type":"ContainerDied","Data":"d79dca5ae6ace8fc217005cb89d787fc07a0787956d3d783c2a45c675e1611f5"}
Jul 28 18:49:32.260512 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:32.260049    2579 scope.go:117] "RemoveContainer" containerID="50147f1b7ae3d8305617a449991b496d0c29d2aeaa659b3082786daf7f67322a"
Jul 28 18:49:32.281979 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:32.281959    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"]
Jul 28 18:49:32.286027 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:32.286003    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-router-scheduler-55c6bcf4c9-vdk4c"]
Jul 28 18:49:32.320300 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:32.320277    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="452ec6b4-e954-428f-b5de-977d0f02ecb3" path="/var/lib/kubelet/pods/452ec6b4-e954-428f-b5de-977d0f02ecb3/volumes"
Jul 28 18:49:34.466830 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:34.466789    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main" containerID="cri-o://133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa" gracePeriod=45
Jul 28 18:49:34.468776 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:34.468737    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main" containerID="cri-o://ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024" gracePeriod=45
Jul 28 18:49:35.286209 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.286164    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="llm-d-routing-sidecar" containerID="cri-o://6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f" gracePeriod=45
Jul 28 18:49:35.433716 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.433690    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:49:35.486072 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.486029    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2hsr\" (UniqueName: \"kubernetes.io/projected/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kube-api-access-t2hsr\") pod \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") "
Jul 28 18:49:35.486586 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.486140    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kserve-provision-location\") pod \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") "
Jul 28 18:49:35.486586 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.486196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tls-certs\") pod \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") "
Jul 28 18:49:35.486586 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.486241    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-home\") pod \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") "
Jul 28 18:49:35.486586 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.486280    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tmp-dir\") pod \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") "
Jul 28 18:49:35.486586 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.486304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-dshm\") pod \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") "
Jul 28 18:49:35.486586 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.486336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-model-cache\") pod \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\" (UID: \"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d\") "
Jul 28 18:49:35.487382 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.486803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" (UID: "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:35.487382 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.486928    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-home" (OuterVolumeSpecName: "home") pod "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" (UID: "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:35.489057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.489010    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" (UID: "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:49:35.489265 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.489231    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kube-api-access-t2hsr" (OuterVolumeSpecName: "kube-api-access-t2hsr") pod "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" (UID: "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d"). InnerVolumeSpecName "kube-api-access-t2hsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:49:35.489368 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.489258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-dshm" (OuterVolumeSpecName: "dshm") pod "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" (UID: "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:35.499471 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.499443    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" (UID: "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:35.544117 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.544096    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:49:35.560786 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.560758    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" (UID: "39f8a118-dc0f-4959-bf50-1ee9a1d07f2d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:35.587942 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.587898    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a885212-29b1-4ba8-a024-cc0bd7c21106-tls-certs\") pod \"3a885212-29b1-4ba8-a024-cc0bd7c21106\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") "
Jul 28 18:49:35.588173 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.587964    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-kserve-provision-location\") pod \"3a885212-29b1-4ba8-a024-cc0bd7c21106\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") "
Jul 28 18:49:35.588173 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588000    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-dshm\") pod \"3a885212-29b1-4ba8-a024-cc0bd7c21106\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") "
Jul 28 18:49:35.588173 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588064    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-tmp-dir\") pod \"3a885212-29b1-4ba8-a024-cc0bd7c21106\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") "
Jul 28 18:49:35.588173 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588123    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-model-cache\") pod \"3a885212-29b1-4ba8-a024-cc0bd7c21106\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") "
Jul 28 18:49:35.588173 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588150    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-home\") pod \"3a885212-29b1-4ba8-a024-cc0bd7c21106\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") "
Jul 28 18:49:35.588486 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588182    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggqk8\" (UniqueName: \"kubernetes.io/projected/3a885212-29b1-4ba8-a024-cc0bd7c21106-kube-api-access-ggqk8\") pod \"3a885212-29b1-4ba8-a024-cc0bd7c21106\" (UID: \"3a885212-29b1-4ba8-a024-cc0bd7c21106\") "
Jul 28 18:49:35.588486 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588463    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.588486 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588483    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.588656 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588497    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.588656 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588509    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.588656 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588522    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.588656 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588537    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2hsr\" (UniqueName: \"kubernetes.io/projected/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kube-api-access-t2hsr\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.588656 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588554    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.588938 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.588914    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-home" (OuterVolumeSpecName: "home") pod "3a885212-29b1-4ba8-a024-cc0bd7c21106" (UID: "3a885212-29b1-4ba8-a024-cc0bd7c21106"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:35.589235 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.589211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a885212-29b1-4ba8-a024-cc0bd7c21106" (UID: "3a885212-29b1-4ba8-a024-cc0bd7c21106"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:35.590638 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.590608    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a885212-29b1-4ba8-a024-cc0bd7c21106-kube-api-access-ggqk8" (OuterVolumeSpecName: "kube-api-access-ggqk8") pod "3a885212-29b1-4ba8-a024-cc0bd7c21106" (UID: "3a885212-29b1-4ba8-a024-cc0bd7c21106"). InnerVolumeSpecName "kube-api-access-ggqk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:49:35.590740 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.590667    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-dshm" (OuterVolumeSpecName: "dshm") pod "3a885212-29b1-4ba8-a024-cc0bd7c21106" (UID: "3a885212-29b1-4ba8-a024-cc0bd7c21106"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:35.590740 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.590705    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a885212-29b1-4ba8-a024-cc0bd7c21106-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a885212-29b1-4ba8-a024-cc0bd7c21106" (UID: "3a885212-29b1-4ba8-a024-cc0bd7c21106"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:49:35.606605 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.606566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a885212-29b1-4ba8-a024-cc0bd7c21106" (UID: "3a885212-29b1-4ba8-a024-cc0bd7c21106"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:35.655448 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.655400    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a885212-29b1-4ba8-a024-cc0bd7c21106" (UID: "3a885212-29b1-4ba8-a024-cc0bd7c21106"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:49:35.690070 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.690038    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.690070 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.690068    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.690265 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.690104    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.690265 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.690116    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.690265 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.690128    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggqk8\" (UniqueName: \"kubernetes.io/projected/3a885212-29b1-4ba8-a024-cc0bd7c21106-kube-api-access-ggqk8\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.690265 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.690136    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a885212-29b1-4ba8-a024-cc0bd7c21106-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:35.690265 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:35.690145    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a885212-29b1-4ba8-a024-cc0bd7c21106-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:49:36.286867 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.286831    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerID="133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa" exitCode=0
Jul 28 18:49:36.286867 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.286858    2579 generic.go:356] "Generic (PLEG): container finished" podID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerID="6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f" exitCode=0
Jul 28 18:49:36.287167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.286888    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"
Jul 28 18:49:36.287167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.286907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" event={"ID":"3a885212-29b1-4ba8-a024-cc0bd7c21106","Type":"ContainerDied","Data":"133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa"}
Jul 28 18:49:36.287167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.286999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" event={"ID":"3a885212-29b1-4ba8-a024-cc0bd7c21106","Type":"ContainerDied","Data":"6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f"}
Jul 28 18:49:36.287167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.287031    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx" event={"ID":"3a885212-29b1-4ba8-a024-cc0bd7c21106","Type":"ContainerDied","Data":"cc34c26bc924ddcb34334baee3d7307edc3ed74510f97c36b8c2f370ddc030a0"}
Jul 28 18:49:36.287167 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.287064    2579 scope.go:117] "RemoveContainer" containerID="133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa"
Jul 28 18:49:36.288626 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.288601    2579 generic.go:356] "Generic (PLEG): container finished" podID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerID="ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024" exitCode=0
Jul 28 18:49:36.288735 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.288687    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"
Jul 28 18:49:36.288735 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.288692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" event={"ID":"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d","Type":"ContainerDied","Data":"ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024"}
Jul 28 18:49:36.288827 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.288744    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm" event={"ID":"39f8a118-dc0f-4959-bf50-1ee9a1d07f2d","Type":"ContainerDied","Data":"cfd4cf7147e14a6310f08b008d0524a119d621ff6f07f78ed7c01db5a0cbe11d"}
Jul 28 18:49:36.296417 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.296397    2579 scope.go:117] "RemoveContainer" containerID="151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d"
Jul 28 18:49:36.305014 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.304999    2579 scope.go:117] "RemoveContainer" containerID="6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f"
Jul 28 18:49:36.311363 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.311342    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"]
Jul 28 18:49:36.313220 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.313201    2579 scope.go:117] "RemoveContainer" containerID="133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa"
Jul 28 18:49:36.313477 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:49:36.313457    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa\": container with ID starting with 133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa not found: ID does not exist" containerID="133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa"
Jul 28 18:49:36.313573 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.313486    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa"} err="failed to get container status \"133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa\": rpc error: code = NotFound desc = could not find container \"133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa\": container with ID starting with 133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa not found: ID does not exist"
Jul 28 18:49:36.313573 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.313504    2579 scope.go:117] "RemoveContainer" containerID="151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d"
Jul 28 18:49:36.313709 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:49:36.313690    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d\": container with ID starting with 151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d not found: ID does not exist" containerID="151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d"
Jul 28 18:49:36.313770 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.313713    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d"} err="failed to get container status \"151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d\": rpc error: code = NotFound desc = could not find container \"151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d\": container with ID starting with 151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d not found: ID does not exist"
Jul 28 18:49:36.313770 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.313730    2579 scope.go:117] "RemoveContainer" containerID="6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f"
Jul 28 18:49:36.313996 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:49:36.313953    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f\": container with ID starting with 6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f not found: ID does not exist" containerID="6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f"
Jul 28 18:49:36.314234 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.314003    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f"} err="failed to get container status \"6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f\": rpc error: code = NotFound desc = could not find container \"6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f\": container with ID starting with 6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f not found: ID does not exist"
Jul 28 18:49:36.314234 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.314024    2579 scope.go:117] "RemoveContainer" containerID="133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa"
Jul 28 18:49:36.315149 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.315117    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa"} err="failed to get container status \"133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa\": rpc error: code = NotFound desc = could not find container \"133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa\": container with ID starting with 133b1320f314ecd798a13781ecfa5a76072bdc8e83f7d11dad3fb62339a649fa not found: ID does not exist"
Jul 28 18:49:36.315149 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.315149    2579 scope.go:117] "RemoveContainer" containerID="151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d"
Jul 28 18:49:36.315437 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.315413    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d"} err="failed to get container status \"151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d\": rpc error: code = NotFound desc = could not find container \"151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d\": container with ID starting with 151594d5b9b134577bc99ac560160c06e5e60d3d4e5b72705a29a410c6a8807d not found: ID does not exist"
Jul 28 18:49:36.315532 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.315439    2579 scope.go:117] "RemoveContainer" containerID="6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f"
Jul 28 18:49:36.315697 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.315677    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f"} err="failed to get container status \"6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f\": rpc error: code = NotFound desc = could not find container \"6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f\": container with ID starting with 6b3e0ca63b1e52ceb62f7042634279ff11c6ddede2e9ac42fe5e44af2acaa52f not found: ID does not exist"
Jul 28 18:49:36.315777 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.315699    2579 scope.go:117] "RemoveContainer" containerID="ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024"
Jul 28 18:49:36.320024 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.320005    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-848584f79dj99gx"]
Jul 28 18:49:36.323783 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.323754    2579 scope.go:117] "RemoveContainer" containerID="eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc"
Jul 28 18:49:36.327394 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.327369    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"]
Jul 28 18:49:36.332791 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.332767    2579 scope.go:117] "RemoveContainer" containerID="ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024"
Jul 28 18:49:36.333057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.333032    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-86j7xwm"]
Jul 28 18:49:36.333057 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:49:36.333043    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024\": container with ID starting with ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024 not found: ID does not exist" containerID="ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024"
Jul 28 18:49:36.333196 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.333066    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024"} err="failed to get container status \"ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024\": rpc error: code = NotFound desc = could not find container \"ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024\": container with ID starting with ec68410d9d19e00eee77c390a8f0c5f49516f93952e27620ae0b81641baa5024 not found: ID does not exist"
Jul 28 18:49:36.333196 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.333101    2579 scope.go:117] "RemoveContainer" containerID="eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc"
Jul 28 18:49:36.333347 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:49:36.333329    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc\": container with ID starting with eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc not found: ID does not exist" containerID="eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc"
Jul 28 18:49:36.333404 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:36.333357    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc"} err="failed to get container status \"eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc\": rpc error: code = NotFound desc = could not find container \"eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc\": container with ID starting with eed070198d44bb7343df75a3b76e46c73eb76f1b02229ee1df7f765f670ec7bc not found: ID does not exist"
Jul 28 18:49:38.320280 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:38.320249    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" path="/var/lib/kubelet/pods/39f8a118-dc0f-4959-bf50-1ee9a1d07f2d/volumes"
Jul 28 18:49:38.320747 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:49:38.320669    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" path="/var/lib/kubelet/pods/3a885212-29b1-4ba8-a024-cc0bd7c21106/volumes"
Jul 28 18:50:04.635949 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.635910    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"]
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636283    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="452ec6b4-e954-428f-b5de-977d0f02ecb3" containerName="main"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636295    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="452ec6b4-e954-428f-b5de-977d0f02ecb3" containerName="main"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636315    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636321    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636330    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636336    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636344    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="storage-initializer"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636349    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="storage-initializer"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636362    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="storage-initializer"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636367    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="storage-initializer"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636375    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="llm-d-routing-sidecar"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636381    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="llm-d-routing-sidecar"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636432    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="452ec6b4-e954-428f-b5de-977d0f02ecb3" containerName="main"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636440    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="llm-d-routing-sidecar"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636445    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a885212-29b1-4ba8-a024-cc0bd7c21106" containerName="main"
Jul 28 18:50:04.636470 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.636457    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f8a118-dc0f-4959-bf50-1ee9a1d07f2d" containerName="main"
Jul 28 18:50:04.639130 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.639112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.641527 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.641502    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Jul 28 18:50:04.641773 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.641757    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-dockercfg-jgrw4\""
Jul 28 18:50:04.641869 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.641840    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"seaweedfs-s3-creds\""
Jul 28 18:50:04.641967 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.641945    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-zxjkm\""
Jul 28 18:50:04.642643 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.642623    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 28 18:50:04.642727 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.642637    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Jul 28 18:50:04.650759 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.650736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-dshm\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.650860 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.650785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31291fd6-0d3d-412e-9227-8809375c59f5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.650860 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.650809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.650949 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.650912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-home\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.650999 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.650948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbsv4\" (UniqueName: \"kubernetes.io/projected/31291fd6-0d3d-412e-9227-8809375c59f5-kube-api-access-jbsv4\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.651058 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.651031    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.651149 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.651068    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.665308 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.665279    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"]
Jul 28 18:50:04.751593 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.751553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.751794 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.751612    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-home\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.751794 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.751630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbsv4\" (UniqueName: \"kubernetes.io/projected/31291fd6-0d3d-412e-9227-8809375c59f5-kube-api-access-jbsv4\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.751794 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.751661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.751794 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.751677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.751794 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.751719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-dshm\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.751794 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.751776    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31291fd6-0d3d-412e-9227-8809375c59f5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.752158 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.752035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.752158 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.752050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-home\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.752158 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.752145    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.752280 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.752177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.754065 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.754040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-dshm\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.754534 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.754512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31291fd6-0d3d-412e-9227-8809375c59f5-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.770373 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.770349    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbsv4\" (UniqueName: \"kubernetes.io/projected/31291fd6-0d3d-412e-9227-8809375c59f5-kube-api-access-jbsv4\") pod \"custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:04.949132 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:04.949035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:05.092009 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:05.091979    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"]
Jul 28 18:50:05.093864 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:50:05.093840    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31291fd6_0d3d_412e_9227_8809375c59f5.slice/crio-aee06f4189e16a8587a4b87969cb50516b48cc1b2e3bdbca8e5f0be87c0b5db6 WatchSource:0}: Error finding container aee06f4189e16a8587a4b87969cb50516b48cc1b2e3bdbca8e5f0be87c0b5db6: Status 404 returned error can't find the container with id aee06f4189e16a8587a4b87969cb50516b48cc1b2e3bdbca8e5f0be87c0b5db6
Jul 28 18:50:05.437806 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:05.437767    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" event={"ID":"31291fd6-0d3d-412e-9227-8809375c59f5","Type":"ContainerStarted","Data":"9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0"}
Jul 28 18:50:05.438030 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:05.437829    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" event={"ID":"31291fd6-0d3d-412e-9227-8809375c59f5","Type":"ContainerStarted","Data":"aee06f4189e16a8587a4b87969cb50516b48cc1b2e3bdbca8e5f0be87c0b5db6"}
Jul 28 18:50:05.438030 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:05.437880    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:06.448261 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:06.448212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" event={"ID":"31291fd6-0d3d-412e-9227-8809375c59f5","Type":"ContainerStarted","Data":"87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec"}
Jul 28 18:50:12.788728 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:12.788695    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:50:12.821093 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:12.821060    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:50:17.467104 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:17.467058    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:34.587697 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:34.587661    2579 generic.go:356] "Generic (PLEG): container finished" podID="31291fd6-0d3d-412e-9227-8809375c59f5" containerID="87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec" exitCode=0
Jul 28 18:50:34.588170 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:34.587738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" event={"ID":"31291fd6-0d3d-412e-9227-8809375c59f5","Type":"ContainerDied","Data":"87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec"}
Jul 28 18:50:35.595683 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:35.595646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" event={"ID":"31291fd6-0d3d-412e-9227-8809375c59f5","Type":"ContainerStarted","Data":"755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074"}
Jul 28 18:50:35.621477 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:35.621414    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podStartSLOduration=3.541510115 podStartE2EDuration="31.621394321s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.079884206s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:50:35.617916041 +0000 UTC m=+6023.847115157" watchObservedRunningTime="2026-07-28 18:50:35.621394321 +0000 UTC m=+6023.850593420"
Jul 28 18:50:44.949214 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:44.949170    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:44.949723 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:44.949256    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:50:44.949803 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:44.949734    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:50:54.950089 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:50:54.950038    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:51:04.950386 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:51:04.950327    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:51:14.950418 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:51:14.950361    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:51:24.949489 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:51:24.949446    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:51:34.950480 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:51:34.950428    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:51:44.949745 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:51:44.949696    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:51:54.949825 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:51:54.949771    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:52:04.949521 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:04.949421    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:52:14.950219 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:14.950163    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:52:24.950507 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:24.950455    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.104:8001/health\": dial tcp 10.132.0.104:8001: connect: connection refused"
Jul 28 18:52:34.959758 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:34.959719    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:52:34.971833 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:34.971804    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:52:43.332772 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:43.332739    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"]
Jul 28 18:52:58.358169 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:58.358098    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main" containerID="cri-o://755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074" gracePeriod=45
Jul 28 18:52:58.973969 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:58.973919    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="llm-d-routing-sidecar" containerID="cri-o://9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0" gracePeriod=45
Jul 28 18:52:59.224914 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.224887    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:52:59.318019 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.317983    2579 generic.go:356] "Generic (PLEG): container finished" podID="31291fd6-0d3d-412e-9227-8809375c59f5" containerID="755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074" exitCode=0
Jul 28 18:52:59.318019 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.318013    2579 generic.go:356] "Generic (PLEG): container finished" podID="31291fd6-0d3d-412e-9227-8809375c59f5" containerID="9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0" exitCode=0
Jul 28 18:52:59.318253 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.318044    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"
Jul 28 18:52:59.318253 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.318057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" event={"ID":"31291fd6-0d3d-412e-9227-8809375c59f5","Type":"ContainerDied","Data":"755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074"}
Jul 28 18:52:59.318253 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.318130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" event={"ID":"31291fd6-0d3d-412e-9227-8809375c59f5","Type":"ContainerDied","Data":"9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0"}
Jul 28 18:52:59.318253 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.318150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh" event={"ID":"31291fd6-0d3d-412e-9227-8809375c59f5","Type":"ContainerDied","Data":"aee06f4189e16a8587a4b87969cb50516b48cc1b2e3bdbca8e5f0be87c0b5db6"}
Jul 28 18:52:59.318253 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.318170    2579 scope.go:117] "RemoveContainer" containerID="755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074"
Jul 28 18:52:59.326841 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.326825    2579 scope.go:117] "RemoveContainer" containerID="87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec"
Jul 28 18:52:59.335215 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.335197    2579 scope.go:117] "RemoveContainer" containerID="9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0"
Jul 28 18:52:59.342576 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.342558    2579 scope.go:117] "RemoveContainer" containerID="755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074"
Jul 28 18:52:59.342808 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:52:59.342791    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074\": container with ID starting with 755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074 not found: ID does not exist" containerID="755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074"
Jul 28 18:52:59.342863 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.342817    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074"} err="failed to get container status \"755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074\": rpc error: code = NotFound desc = could not find container \"755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074\": container with ID starting with 755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074 not found: ID does not exist"
Jul 28 18:52:59.342863 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.342834    2579 scope.go:117] "RemoveContainer" containerID="87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec"
Jul 28 18:52:59.343093 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:52:59.343057    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec\": container with ID starting with 87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec not found: ID does not exist" containerID="87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec"
Jul 28 18:52:59.343142 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.343096    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec"} err="failed to get container status \"87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec\": rpc error: code = NotFound desc = could not find container \"87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec\": container with ID starting with 87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec not found: ID does not exist"
Jul 28 18:52:59.343142 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.343119    2579 scope.go:117] "RemoveContainer" containerID="9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0"
Jul 28 18:52:59.343336 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:52:59.343312    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0\": container with ID starting with 9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0 not found: ID does not exist" containerID="9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0"
Jul 28 18:52:59.343384 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.343338    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0"} err="failed to get container status \"9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0\": rpc error: code = NotFound desc = could not find container \"9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0\": container with ID starting with 9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0 not found: ID does not exist"
Jul 28 18:52:59.343384 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.343351    2579 scope.go:117] "RemoveContainer" containerID="755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074"
Jul 28 18:52:59.343536 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.343520    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074"} err="failed to get container status \"755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074\": rpc error: code = NotFound desc = could not find container \"755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074\": container with ID starting with 755eea58554b2fb76dad613f99994eccd48fe85f1b6d4bd57d0beef8d8c4d074 not found: ID does not exist"
Jul 28 18:52:59.343536 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.343535    2579 scope.go:117] "RemoveContainer" containerID="87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec"
Jul 28 18:52:59.343726 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.343712    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec"} err="failed to get container status \"87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec\": rpc error: code = NotFound desc = could not find container \"87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec\": container with ID starting with 87aeee3de55d94c1a4de85447782c45ef8bd30041006ae503dcf4ab6db414cec not found: ID does not exist"
Jul 28 18:52:59.343767 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.343725    2579 scope.go:117] "RemoveContainer" containerID="9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0"
Jul 28 18:52:59.343885 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.343870    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0"} err="failed to get container status \"9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0\": rpc error: code = NotFound desc = could not find container \"9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0\": container with ID starting with 9da7ce7f29771ac64cfb382b186c3ac28efba95ef9faa39c4fe51cefa36879e0 not found: ID does not exist"
Jul 28 18:52:59.368252 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.368231    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbsv4\" (UniqueName: \"kubernetes.io/projected/31291fd6-0d3d-412e-9227-8809375c59f5-kube-api-access-jbsv4\") pod \"31291fd6-0d3d-412e-9227-8809375c59f5\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") "
Jul 28 18:52:59.368624 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.368272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-home\") pod \"31291fd6-0d3d-412e-9227-8809375c59f5\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") "
Jul 28 18:52:59.368624 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.368306    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-kserve-provision-location\") pod \"31291fd6-0d3d-412e-9227-8809375c59f5\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") "
Jul 28 18:52:59.368624 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.368341    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-dshm\") pod \"31291fd6-0d3d-412e-9227-8809375c59f5\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") "
Jul 28 18:52:59.368624 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.368376    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31291fd6-0d3d-412e-9227-8809375c59f5-tls-certs\") pod \"31291fd6-0d3d-412e-9227-8809375c59f5\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") "
Jul 28 18:52:59.368624 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.368397    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-tmp-dir\") pod \"31291fd6-0d3d-412e-9227-8809375c59f5\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") "
Jul 28 18:52:59.368624 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.368485    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-model-cache\") pod \"31291fd6-0d3d-412e-9227-8809375c59f5\" (UID: \"31291fd6-0d3d-412e-9227-8809375c59f5\") "
Jul 28 18:52:59.368987 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.368967    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "31291fd6-0d3d-412e-9227-8809375c59f5" (UID: "31291fd6-0d3d-412e-9227-8809375c59f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:52:59.369041 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.368990    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-home" (OuterVolumeSpecName: "home") pod "31291fd6-0d3d-412e-9227-8809375c59f5" (UID: "31291fd6-0d3d-412e-9227-8809375c59f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:52:59.370570 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.370544    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31291fd6-0d3d-412e-9227-8809375c59f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31291fd6-0d3d-412e-9227-8809375c59f5" (UID: "31291fd6-0d3d-412e-9227-8809375c59f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:52:59.370661 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.370603    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31291fd6-0d3d-412e-9227-8809375c59f5-kube-api-access-jbsv4" (OuterVolumeSpecName: "kube-api-access-jbsv4") pod "31291fd6-0d3d-412e-9227-8809375c59f5" (UID: "31291fd6-0d3d-412e-9227-8809375c59f5"). InnerVolumeSpecName "kube-api-access-jbsv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:52:59.370734 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.370719    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-dshm" (OuterVolumeSpecName: "dshm") pod "31291fd6-0d3d-412e-9227-8809375c59f5" (UID: "31291fd6-0d3d-412e-9227-8809375c59f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:52:59.381256 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.381227    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31291fd6-0d3d-412e-9227-8809375c59f5" (UID: "31291fd6-0d3d-412e-9227-8809375c59f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:52:59.438783 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.438698    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31291fd6-0d3d-412e-9227-8809375c59f5" (UID: "31291fd6-0d3d-412e-9227-8809375c59f5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:52:59.469995 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.469970    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.469995 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.469991    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.469995 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.470001    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.470234 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.470010    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31291fd6-0d3d-412e-9227-8809375c59f5-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.470234 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.470019    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.470234 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.470027    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31291fd6-0d3d-412e-9227-8809375c59f5-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.470234 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.470035    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbsv4\" (UniqueName: \"kubernetes.io/projected/31291fd6-0d3d-412e-9227-8809375c59f5-kube-api-access-jbsv4\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:52:59.642063 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.642029    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"]
Jul 28 18:52:59.648746 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:52:59.648714    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-6ff4cbdcc7-nxchh"]
Jul 28 18:53:00.320591 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:00.320555    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" path="/var/lib/kubelet/pods/31291fd6-0d3d-412e-9227-8809375c59f5/volumes"
Jul 28 18:53:26.059111 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.059063    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"]
Jul 28 18:53:26.059670 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.059457    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main"
Jul 28 18:53:26.059670 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.059469    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main"
Jul 28 18:53:26.059670 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.059491    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="storage-initializer"
Jul 28 18:53:26.059670 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.059497    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="storage-initializer"
Jul 28 18:53:26.059670 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.059506    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="llm-d-routing-sidecar"
Jul 28 18:53:26.059670 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.059511    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="llm-d-routing-sidecar"
Jul 28 18:53:26.059670 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.059568    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="llm-d-routing-sidecar"
Jul 28 18:53:26.059670 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.059579    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="31291fd6-0d3d-412e-9227-8809375c59f5" containerName="main"
Jul 28 18:53:26.062963 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.062941    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.066263 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.065973    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Jul 28 18:53:26.066263 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.066028    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-dockercfg-mz7v7\""
Jul 28 18:53:26.066467 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.066303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"seaweedfs-s3-creds\""
Jul 28 18:53:26.066467 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.066322    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Jul 28 18:53:26.066467 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.066336    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 28 18:53:26.067239 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.067219    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-rr7zn\""
Jul 28 18:53:26.079466 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.079440    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"]
Jul 28 18:53:26.208877 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.208836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-dshm\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.209057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.208887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-home\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.209057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.208925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tls-certs\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.209057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.208956    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gckfp\" (UniqueName: \"kubernetes.io/projected/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kube-api-access-gckfp\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.209057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.208989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tmp-dir\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.209057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.209013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.209057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.209044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-model-cache\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.310556 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.310474    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tmp-dir\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.310556 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.310516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.310791 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.310561    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-model-cache\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.310791 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.310591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-dshm\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.310791 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.310624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-home\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.310791 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.310669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tls-certs\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.310791 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.310687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gckfp\" (UniqueName: \"kubernetes.io/projected/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kube-api-access-gckfp\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.311063 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.310891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tmp-dir\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.311063 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.310974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.311063 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.310998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-model-cache\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.311063 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.311015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-home\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.313178 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.313153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-dshm\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.313904 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.313875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tls-certs\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.326005 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.325975    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gckfp\" (UniqueName: \"kubernetes.io/projected/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kube-api-access-gckfp\") pod \"router-with-refs-pd-test-kserve-75d6866649-smbnv\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.374161 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.374125    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:26.516338 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.516309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"]
Jul 28 18:53:26.518671 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:53:26.518642    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa45cb5c_e4e2_4a8c_9261_28295e3257d8.slice/crio-e15b2de28b57fe5bd2484236914150009addfe172648e541996e0baad36c05d6 WatchSource:0}: Error finding container e15b2de28b57fe5bd2484236914150009addfe172648e541996e0baad36c05d6: Status 404 returned error can't find the container with id e15b2de28b57fe5bd2484236914150009addfe172648e541996e0baad36c05d6
Jul 28 18:53:26.685169 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.685126    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"]
Jul 28 18:53:26.688669 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.688643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:53:26.691223 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.691150    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-6nrlv\""
Jul 28 18:53:26.697678 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.697650    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"]
Jul 28 18:53:26.815684 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.815647    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4tt\" (UniqueName: \"kubernetes.io/projected/82b08733-c20d-44fd-a94e-860d28a733e3-kube-api-access-8c4tt\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr\" (UID: \"82b08733-c20d-44fd-a94e-860d28a733e3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:53:26.815872 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.815694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b08733-c20d-44fd-a94e-860d28a733e3-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr\" (UID: \"82b08733-c20d-44fd-a94e-860d28a733e3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:53:26.917098 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.916990    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4tt\" (UniqueName: \"kubernetes.io/projected/82b08733-c20d-44fd-a94e-860d28a733e3-kube-api-access-8c4tt\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr\" (UID: \"82b08733-c20d-44fd-a94e-860d28a733e3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:53:26.917098 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.917038    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b08733-c20d-44fd-a94e-860d28a733e3-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr\" (UID: \"82b08733-c20d-44fd-a94e-860d28a733e3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:53:26.919503 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.919476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b08733-c20d-44fd-a94e-860d28a733e3-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr\" (UID: \"82b08733-c20d-44fd-a94e-860d28a733e3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:53:26.925494 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.925471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4tt\" (UniqueName: \"kubernetes.io/projected/82b08733-c20d-44fd-a94e-860d28a733e3-kube-api-access-8c4tt\") pod \"router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr\" (UID: \"82b08733-c20d-44fd-a94e-860d28a733e3\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:53:26.999809 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:26.999773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:53:27.133629 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:27.133594    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"]
Jul 28 18:53:27.137242 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:53:27.137216    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b08733_c20d_44fd_a94e_860d28a733e3.slice/crio-d23c2bee0be4435e99e62ec01e448fa7af45d1ffdbf686087e36d0a20621671b WatchSource:0}: Error finding container d23c2bee0be4435e99e62ec01e448fa7af45d1ffdbf686087e36d0a20621671b: Status 404 returned error can't find the container with id d23c2bee0be4435e99e62ec01e448fa7af45d1ffdbf686087e36d0a20621671b
Jul 28 18:53:27.454845 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:27.454748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" event={"ID":"fa45cb5c-e4e2-4a8c-9261-28295e3257d8","Type":"ContainerStarted","Data":"75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9"}
Jul 28 18:53:27.454845 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:27.454810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" event={"ID":"fa45cb5c-e4e2-4a8c-9261-28295e3257d8","Type":"ContainerStarted","Data":"e15b2de28b57fe5bd2484236914150009addfe172648e541996e0baad36c05d6"}
Jul 28 18:53:27.454845 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:27.454841    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:27.456518 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:27.456489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr" event={"ID":"82b08733-c20d-44fd-a94e-860d28a733e3","Type":"ContainerStarted","Data":"11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e"}
Jul 28 18:53:27.456650 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:27.456536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr" event={"ID":"82b08733-c20d-44fd-a94e-860d28a733e3","Type":"ContainerStarted","Data":"d23c2bee0be4435e99e62ec01e448fa7af45d1ffdbf686087e36d0a20621671b"}
Jul 28 18:53:27.456650 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:27.456557    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:53:27.493671 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:27.493623    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr" podStartSLOduration=1.49360278 podStartE2EDuration="1.49360278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:27.49256687 +0000 UTC m=+6195.721765968" watchObservedRunningTime="2026-07-28 18:53:27.49360278 +0000 UTC m=+6195.722801878"
Jul 28 18:53:28.463670 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:28.463627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" event={"ID":"fa45cb5c-e4e2-4a8c-9261-28295e3257d8","Type":"ContainerStarted","Data":"2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef"}
Jul 28 18:53:35.502614 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:35.502578    2579 generic.go:356] "Generic (PLEG): container finished" podID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerID="2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef" exitCode=0
Jul 28 18:53:35.503030 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:35.502648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" event={"ID":"fa45cb5c-e4e2-4a8c-9261-28295e3257d8","Type":"ContainerDied","Data":"2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef"}
Jul 28 18:53:36.510508 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:36.510475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" event={"ID":"fa45cb5c-e4e2-4a8c-9261-28295e3257d8","Type":"ContainerStarted","Data":"a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d"}
Jul 28 18:53:36.523009 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:36.522979    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:36.539755 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:36.539692    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podStartSLOduration=3.046550773 podStartE2EDuration="10.539671566s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.493120793s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:36.535785769 +0000 UTC m=+6204.764984868" watchObservedRunningTime="2026-07-28 18:53:36.539671566 +0000 UTC m=+6204.768870666"
Jul 28 18:53:46.374724 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:46.374688    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:46.375266 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:46.374854    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:53:46.375527 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:46.375499    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:53:56.375050 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:56.374995    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:53:58.465118 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:53:58.465088    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:54:06.375591 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:54:06.375538    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:54:16.374952 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:54:16.374898    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:54:26.374725 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:54:26.374671    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:54:36.375612 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:54:36.375566    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:54:46.374834 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:54:46.374784    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:54:56.375420 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:54:56.375369    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:55:06.375412 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:06.375301    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:55:12.824998 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:12.824969    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:55:12.855369 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:12.855341    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:55:16.375381 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:16.375331    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:55:26.374813 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:26.374734    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.105:8001/health\": dial tcp 10.132.0.105:8001: connect: connection refused"
Jul 28 18:55:36.390508 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:36.390470    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:55:36.402321 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:36.402297    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:55:45.340060 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:45.340022    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"]
Jul 28 18:55:45.349381 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:45.349351    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"]
Jul 28 18:55:48.464996 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:48.464951    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr" podUID="82b08733-c20d-44fd-a94e-860d28a733e3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:55:58.464985 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:55:58.464947    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr" podUID="82b08733-c20d-44fd-a94e-860d28a733e3" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 28 18:56:00.361581 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:00.361543    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr" podUID="82b08733-c20d-44fd-a94e-860d28a733e3" containerName="main" containerID="cri-o://11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e" gracePeriod=45
Jul 28 18:56:00.371599 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:00.371565    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main" containerID="cri-o://a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d" gracePeriod=45
Jul 28 18:56:00.622711 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:00.622631    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:56:00.684614 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:00.684580    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c4tt\" (UniqueName: \"kubernetes.io/projected/82b08733-c20d-44fd-a94e-860d28a733e3-kube-api-access-8c4tt\") pod \"82b08733-c20d-44fd-a94e-860d28a733e3\" (UID: \"82b08733-c20d-44fd-a94e-860d28a733e3\") "
Jul 28 18:56:00.684814 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:00.684645    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b08733-c20d-44fd-a94e-860d28a733e3-tls-certs\") pod \"82b08733-c20d-44fd-a94e-860d28a733e3\" (UID: \"82b08733-c20d-44fd-a94e-860d28a733e3\") "
Jul 28 18:56:00.686865 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:00.686838    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b08733-c20d-44fd-a94e-860d28a733e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "82b08733-c20d-44fd-a94e-860d28a733e3" (UID: "82b08733-c20d-44fd-a94e-860d28a733e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:56:00.687000 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:00.686905    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b08733-c20d-44fd-a94e-860d28a733e3-kube-api-access-8c4tt" (OuterVolumeSpecName: "kube-api-access-8c4tt") pod "82b08733-c20d-44fd-a94e-860d28a733e3" (UID: "82b08733-c20d-44fd-a94e-860d28a733e3"). InnerVolumeSpecName "kube-api-access-8c4tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:00.786497 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:00.786451    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4tt\" (UniqueName: \"kubernetes.io/projected/82b08733-c20d-44fd-a94e-860d28a733e3-kube-api-access-8c4tt\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:00.786497 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:00.786487    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b08733-c20d-44fd-a94e-860d28a733e3-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:00.983279 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:00.983246    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="llm-d-routing-sidecar" containerID="cri-o://75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9" gracePeriod=45
Jul 28 18:56:01.231327 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.231292    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:56:01.259725 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.259693    2579 generic.go:356] "Generic (PLEG): container finished" podID="82b08733-c20d-44fd-a94e-860d28a733e3" containerID="11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e" exitCode=0
Jul 28 18:56:01.259907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.259750    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"
Jul 28 18:56:01.259907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.259775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr" event={"ID":"82b08733-c20d-44fd-a94e-860d28a733e3","Type":"ContainerDied","Data":"11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e"}
Jul 28 18:56:01.259907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.259864    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr" event={"ID":"82b08733-c20d-44fd-a94e-860d28a733e3","Type":"ContainerDied","Data":"d23c2bee0be4435e99e62ec01e448fa7af45d1ffdbf686087e36d0a20621671b"}
Jul 28 18:56:01.259907 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.259902    2579 scope.go:117] "RemoveContainer" containerID="11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e"
Jul 28 18:56:01.262203 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.262174    2579 generic.go:356] "Generic (PLEG): container finished" podID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerID="a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d" exitCode=0
Jul 28 18:56:01.262203 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.262200    2579 generic.go:356] "Generic (PLEG): container finished" podID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerID="75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9" exitCode=0
Jul 28 18:56:01.262397 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.262242    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"
Jul 28 18:56:01.262397 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.262253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" event={"ID":"fa45cb5c-e4e2-4a8c-9261-28295e3257d8","Type":"ContainerDied","Data":"a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d"}
Jul 28 18:56:01.262397 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.262306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" event={"ID":"fa45cb5c-e4e2-4a8c-9261-28295e3257d8","Type":"ContainerDied","Data":"75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9"}
Jul 28 18:56:01.262397 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.262331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv" event={"ID":"fa45cb5c-e4e2-4a8c-9261-28295e3257d8","Type":"ContainerDied","Data":"e15b2de28b57fe5bd2484236914150009addfe172648e541996e0baad36c05d6"}
Jul 28 18:56:01.270807 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.270791    2579 scope.go:117] "RemoveContainer" containerID="11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e"
Jul 28 18:56:01.271191 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:56:01.271046    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e\": container with ID starting with 11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e not found: ID does not exist" containerID="11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e"
Jul 28 18:56:01.271191 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.271097    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e"} err="failed to get container status \"11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e\": rpc error: code = NotFound desc = could not find container \"11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e\": container with ID starting with 11a73d862a84a52d307252e95abc0dc013490c058c192c4588c9b6756dae2e8e not found: ID does not exist"
Jul 28 18:56:01.271191 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.271121    2579 scope.go:117] "RemoveContainer" containerID="a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d"
Jul 28 18:56:01.279164 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.279140    2579 scope.go:117] "RemoveContainer" containerID="2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef"
Jul 28 18:56:01.283063 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.283041    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"]
Jul 28 18:56:01.288296 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.288272    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-7976968ffh49tr"]
Jul 28 18:56:01.289707 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.289688    2579 scope.go:117] "RemoveContainer" containerID="75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9"
Jul 28 18:56:01.292775 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.292755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-home\") pod \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") "
Jul 28 18:56:01.292881 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.292806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-dshm\") pod \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") "
Jul 28 18:56:01.292881 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.292865    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tls-certs\") pod \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") "
Jul 28 18:56:01.292999 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.292887    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kserve-provision-location\") pod \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") "
Jul 28 18:56:01.292999 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.292928    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-model-cache\") pod \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") "
Jul 28 18:56:01.292999 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.292980    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gckfp\" (UniqueName: \"kubernetes.io/projected/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kube-api-access-gckfp\") pod \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") "
Jul 28 18:56:01.293198 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.293005    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tmp-dir\") pod \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\" (UID: \"fa45cb5c-e4e2-4a8c-9261-28295e3257d8\") "
Jul 28 18:56:01.295091 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.293455    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-home" (OuterVolumeSpecName: "home") pod "fa45cb5c-e4e2-4a8c-9261-28295e3257d8" (UID: "fa45cb5c-e4e2-4a8c-9261-28295e3257d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:01.295091 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.293478    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa45cb5c-e4e2-4a8c-9261-28295e3257d8" (UID: "fa45cb5c-e4e2-4a8c-9261-28295e3257d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:01.296104 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.296050    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-dshm" (OuterVolumeSpecName: "dshm") pod "fa45cb5c-e4e2-4a8c-9261-28295e3257d8" (UID: "fa45cb5c-e4e2-4a8c-9261-28295e3257d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:01.296210 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.296130    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa45cb5c-e4e2-4a8c-9261-28295e3257d8" (UID: "fa45cb5c-e4e2-4a8c-9261-28295e3257d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:56:01.296298 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.296276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kube-api-access-gckfp" (OuterVolumeSpecName: "kube-api-access-gckfp") pod "fa45cb5c-e4e2-4a8c-9261-28295e3257d8" (UID: "fa45cb5c-e4e2-4a8c-9261-28295e3257d8"). InnerVolumeSpecName "kube-api-access-gckfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:01.298306 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.298289    2579 scope.go:117] "RemoveContainer" containerID="a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d"
Jul 28 18:56:01.298617 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:56:01.298593    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d\": container with ID starting with a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d not found: ID does not exist" containerID="a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d"
Jul 28 18:56:01.298666 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.298632    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d"} err="failed to get container status \"a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d\": rpc error: code = NotFound desc = could not find container \"a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d\": container with ID starting with a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d not found: ID does not exist"
Jul 28 18:56:01.298666 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.298660    2579 scope.go:117] "RemoveContainer" containerID="2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef"
Jul 28 18:56:01.298907 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:56:01.298888    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef\": container with ID starting with 2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef not found: ID does not exist" containerID="2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef"
Jul 28 18:56:01.298984 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.298915    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef"} err="failed to get container status \"2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef\": rpc error: code = NotFound desc = could not find container \"2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef\": container with ID starting with 2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef not found: ID does not exist"
Jul 28 18:56:01.298984 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.298937    2579 scope.go:117] "RemoveContainer" containerID="75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9"
Jul 28 18:56:01.299215 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:56:01.299192    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9\": container with ID starting with 75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9 not found: ID does not exist" containerID="75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9"
Jul 28 18:56:01.299268 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.299220    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9"} err="failed to get container status \"75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9\": rpc error: code = NotFound desc = could not find container \"75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9\": container with ID starting with 75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9 not found: ID does not exist"
Jul 28 18:56:01.299268 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.299240    2579 scope.go:117] "RemoveContainer" containerID="a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d"
Jul 28 18:56:01.299487 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.299468    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d"} err="failed to get container status \"a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d\": rpc error: code = NotFound desc = could not find container \"a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d\": container with ID starting with a237166390169fdbf2d0d893f635bf890e5a8a47344b474fc4913f1c5de7767d not found: ID does not exist"
Jul 28 18:56:01.299567 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.299489    2579 scope.go:117] "RemoveContainer" containerID="2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef"
Jul 28 18:56:01.299760 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.299743    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef"} err="failed to get container status \"2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef\": rpc error: code = NotFound desc = could not find container \"2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef\": container with ID starting with 2f4ad7a1288f473c1f5e7080e5953883d29ebbd9c3910f8cd815a21bd6b665ef not found: ID does not exist"
Jul 28 18:56:01.299830 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.299761    2579 scope.go:117] "RemoveContainer" containerID="75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9"
Jul 28 18:56:01.299972 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.299955    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9"} err="failed to get container status \"75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9\": rpc error: code = NotFound desc = could not find container \"75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9\": container with ID starting with 75ca4ab911c89f95e6b7c48cfbc615f79940c903d58e35e94f313250a8a824e9 not found: ID does not exist"
Jul 28 18:56:01.307169 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.307146    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa45cb5c-e4e2-4a8c-9261-28295e3257d8" (UID: "fa45cb5c-e4e2-4a8c-9261-28295e3257d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:01.360366 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.360333    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa45cb5c-e4e2-4a8c-9261-28295e3257d8" (UID: "fa45cb5c-e4e2-4a8c-9261-28295e3257d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:01.394272 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.394243    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tls-certs\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.394272 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.394273    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kserve-provision-location\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.394632 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.394283    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-model-cache\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.394632 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.394292    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gckfp\" (UniqueName: \"kubernetes.io/projected/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-kube-api-access-gckfp\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.394632 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.394300    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-tmp-dir\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.394632 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.394310    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-home\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.394632 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.394317    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa45cb5c-e4e2-4a8c-9261-28295e3257d8-dshm\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:01.587966 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.587932    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"]
Jul 28 18:56:01.590750 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:01.590725    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-75d6866649-smbnv"]
Jul 28 18:56:02.320240 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:02.320205    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82b08733-c20d-44fd-a94e-860d28a733e3" path="/var/lib/kubelet/pods/82b08733-c20d-44fd-a94e-860d28a733e3/volumes"
Jul 28 18:56:02.320800 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:02.320743    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" path="/var/lib/kubelet/pods/fa45cb5c-e4e2-4a8c-9261-28295e3257d8/volumes"
Jul 28 18:56:08.102715 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.102673    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-867wq/must-gather-pjdrd"]
Jul 28 18:56:08.103371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103248    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="storage-initializer"
Jul 28 18:56:08.103371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103269    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="storage-initializer"
Jul 28 18:56:08.103371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103303    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82b08733-c20d-44fd-a94e-860d28a733e3" containerName="main"
Jul 28 18:56:08.103371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103321    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="82b08733-c20d-44fd-a94e-860d28a733e3" containerName="main"
Jul 28 18:56:08.103371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103332    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main"
Jul 28 18:56:08.103371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103341    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main"
Jul 28 18:56:08.103371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103354    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="llm-d-routing-sidecar"
Jul 28 18:56:08.103371 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103361    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="llm-d-routing-sidecar"
Jul 28 18:56:08.103800 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103441    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="main"
Jul 28 18:56:08.103800 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103452    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="82b08733-c20d-44fd-a94e-860d28a733e3" containerName="main"
Jul 28 18:56:08.103800 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.103470    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa45cb5c-e4e2-4a8c-9261-28295e3257d8" containerName="llm-d-routing-sidecar"
Jul 28 18:56:08.107509 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.107487    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:08.110316 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.110282    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-867wq\"/\"kube-root-ca.crt\""
Jul 28 18:56:08.110316 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.110308    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-867wq\"/\"openshift-service-ca.crt\""
Jul 28 18:56:08.110458 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.110297    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-867wq\"/\"default-dockercfg-sbbht\""
Jul 28 18:56:08.121519 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.121497    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-867wq/must-gather-pjdrd"]
Jul 28 18:56:08.258482 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.258452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zxn4\" (UniqueName: \"kubernetes.io/projected/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-kube-api-access-6zxn4\") pod \"must-gather-pjdrd\" (UID: \"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435\") " pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:08.258647 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.258501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-must-gather-output\") pod \"must-gather-pjdrd\" (UID: \"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435\") " pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:08.359287 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.359194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zxn4\" (UniqueName: \"kubernetes.io/projected/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-kube-api-access-6zxn4\") pod \"must-gather-pjdrd\" (UID: \"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435\") " pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:08.359287 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.359262    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-must-gather-output\") pod \"must-gather-pjdrd\" (UID: \"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435\") " pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:08.359616 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.359594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-must-gather-output\") pod \"must-gather-pjdrd\" (UID: \"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435\") " pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:08.367335 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.367300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zxn4\" (UniqueName: \"kubernetes.io/projected/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-kube-api-access-6zxn4\") pod \"must-gather-pjdrd\" (UID: \"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435\") " pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:08.417377 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.417335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:08.550502 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.550475    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-867wq/must-gather-pjdrd"]
Jul 28 18:56:08.552522 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:56:08.552484    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode624ca03_3ad2_4320_8fa0_6d9dd8fd0435.slice/crio-bf1b48df421478101a5f9adf0bd41d0b68c0f7e6a5791076ed0b360568de63ef WatchSource:0}: Error finding container bf1b48df421478101a5f9adf0bd41d0b68c0f7e6a5791076ed0b360568de63ef: Status 404 returned error can't find the container with id bf1b48df421478101a5f9adf0bd41d0b68c0f7e6a5791076ed0b360568de63ef
Jul 28 18:56:08.554588 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:08.554570    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:56:09.308547 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:09.308478    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-867wq/must-gather-pjdrd" event={"ID":"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435","Type":"ContainerStarted","Data":"bf1b48df421478101a5f9adf0bd41d0b68c0f7e6a5791076ed0b360568de63ef"}
Jul 28 18:56:15.349632 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:15.349538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-867wq/must-gather-pjdrd" event={"ID":"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435","Type":"ContainerStarted","Data":"3f4a5fb4674d3e2e91226495ecfc4fef08b2c6ce48f94d3e7b24eefd86ab7c8d"}
Jul 28 18:56:15.349632 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:15.349590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-867wq/must-gather-pjdrd" event={"ID":"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435","Type":"ContainerStarted","Data":"be4c5341da20c5d5fe69b34251c709ed27ba8873b47043d94bfba656a76b6765"}
Jul 28 18:56:15.364603 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:15.364557    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-867wq/must-gather-pjdrd" podStartSLOduration=1.094719269 podStartE2EDuration="7.364543674s" totalImagesPullingTime="6.269824405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:56:15.364106565 +0000 UTC m=+6363.593305670" watchObservedRunningTime="2026-07-28 18:56:15.364543674 +0000 UTC m=+6363.593742797"
Jul 28 18:56:31.508577 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:31.508547    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dc8cf7cff-gn2tw_e77175ad-f991-48f4-bfa2-b35bdbcb138e/router/0.log"
Jul 28 18:56:32.579553 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:32.579520    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dc8cf7cff-gn2tw_e77175ad-f991-48f4-bfa2-b35bdbcb138e/router/0.log"
Jul 28 18:56:33.874472 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:33.874442    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dc8cf7cff-gn2tw_e77175ad-f991-48f4-bfa2-b35bdbcb138e/router/0.log"
Jul 28 18:56:35.044319 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:35.044283    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dc8cf7cff-gn2tw_e77175ad-f991-48f4-bfa2-b35bdbcb138e/router/0.log"
Jul 28 18:56:36.186818 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:36.186787    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:37.387589 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:37.387557    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:38.548680 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:38.548645    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:39.754863 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:39.754821    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:40.916867 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:40.916833    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:42.063439 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:42.063389    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:43.233102 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:43.233032    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:44.398833 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:44.398801    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:45.570782 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:45.570753    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:46.761627 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:46.761593    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:47.944428 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:47.944393    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:49.123832 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:49.123800    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:50.351024 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:50.350989    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:56:51.548562 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:51.548525    2579 generic.go:356] "Generic (PLEG): container finished" podID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" containerID="be4c5341da20c5d5fe69b34251c709ed27ba8873b47043d94bfba656a76b6765" exitCode=0
Jul 28 18:56:51.549028 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:51.548597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-867wq/must-gather-pjdrd" event={"ID":"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435","Type":"ContainerDied","Data":"be4c5341da20c5d5fe69b34251c709ed27ba8873b47043d94bfba656a76b6765"}
Jul 28 18:56:51.549028 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:51.548936    2579 scope.go:117] "RemoveContainer" containerID="be4c5341da20c5d5fe69b34251c709ed27ba8873b47043d94bfba656a76b6765"
Jul 28 18:56:51.902507 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:51.902463    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-867wq_must-gather-pjdrd_e624ca03-3ad2-4320-8fa0-6d9dd8fd0435/gather/0.log"
Jul 28 18:56:55.200830 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:55.200797    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x8kpv_29dafc6b-d587-47f3-9ea9-f10cacb103fe/global-pull-secret-syncer/0.log"
Jul 28 18:56:55.317673 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:55.317643    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f8nqb_3e9a6f3f-f55c-471c-8678-b604feccff48/konnectivity-agent/0.log"
Jul 28 18:56:55.430057 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:55.430023    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-111.ec2.internal_cb470419a847df4b595db3785a098356/haproxy/0.log"
Jul 28 18:56:57.357652 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.357618    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-867wq/must-gather-pjdrd"]
Jul 28 18:56:57.358098 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.357776    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-867wq/must-gather-pjdrd" podUID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" containerName="copy" containerID="cri-o://3f4a5fb4674d3e2e91226495ecfc4fef08b2c6ce48f94d3e7b24eefd86ab7c8d" gracePeriod=2
Jul 28 18:56:57.360141 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:56:57.360101    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pjdrd\" is forbidden: User \"system:node:ip-10-0-136-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-867wq\": no relationship found between node 'ip-10-0-136-111.ec2.internal' and this object" podUID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:57.361419 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.361394    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-867wq/must-gather-pjdrd"]
Jul 28 18:56:57.582042 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.582020    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-867wq_must-gather-pjdrd_e624ca03-3ad2-4320-8fa0-6d9dd8fd0435/copy/0.log"
Jul 28 18:56:57.582438 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.582416    2579 generic.go:356] "Generic (PLEG): container finished" podID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" containerID="3f4a5fb4674d3e2e91226495ecfc4fef08b2c6ce48f94d3e7b24eefd86ab7c8d" exitCode=143
Jul 28 18:56:57.596299 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.596282    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-867wq_must-gather-pjdrd_e624ca03-3ad2-4320-8fa0-6d9dd8fd0435/copy/0.log"
Jul 28 18:56:57.597883 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.597869    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:57.600182 ip-10-0-136-111 kubenswrapper[2579]: E0728 18:56:57.600162    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-pjdrd\" is forbidden: User \"system:node:ip-10-0-136-111.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-867wq\": no relationship found between node 'ip-10-0-136-111.ec2.internal' and this object" podUID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:57.718874 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.718792    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-must-gather-output\") pod \"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435\" (UID: \"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435\") "
Jul 28 18:56:57.719031 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.718911    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zxn4\" (UniqueName: \"kubernetes.io/projected/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-kube-api-access-6zxn4\") pod \"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435\" (UID: \"e624ca03-3ad2-4320-8fa0-6d9dd8fd0435\") "
Jul 28 18:56:57.720953 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.720922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-kube-api-access-6zxn4" (OuterVolumeSpecName: "kube-api-access-6zxn4") pod "e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" (UID: "e624ca03-3ad2-4320-8fa0-6d9dd8fd0435"). InnerVolumeSpecName "kube-api-access-6zxn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:56:57.726743 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.726710    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" (UID: "e624ca03-3ad2-4320-8fa0-6d9dd8fd0435"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 18:56:57.819653 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.819608    2579 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-must-gather-output\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:57.819653 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:57.819647    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zxn4\" (UniqueName: \"kubernetes.io/projected/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435-kube-api-access-6zxn4\") on node \"ip-10-0-136-111.ec2.internal\" DevicePath \"\""
Jul 28 18:56:58.319803 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:58.319769    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" path="/var/lib/kubelet/pods/e624ca03-3ad2-4320-8fa0-6d9dd8fd0435/volumes"
Jul 28 18:56:58.588715 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:58.588639    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-867wq_must-gather-pjdrd_e624ca03-3ad2-4320-8fa0-6d9dd8fd0435/copy/0.log"
Jul 28 18:56:58.589182 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:58.589045    2579 scope.go:117] "RemoveContainer" containerID="3f4a5fb4674d3e2e91226495ecfc4fef08b2c6ce48f94d3e7b24eefd86ab7c8d"
Jul 28 18:56:58.589182 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:58.589070    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-867wq/must-gather-pjdrd"
Jul 28 18:56:58.597142 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:56:58.597124    2579 scope.go:117] "RemoveContainer" containerID="be4c5341da20c5d5fe69b34251c709ed27ba8873b47043d94bfba656a76b6765"
Jul 28 18:57:00.919939 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:00.919902    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-l7cg9_58bffa62-0706-48f1-b219-ab028f6934d5/manager/0.log"
Jul 28 18:57:02.304977 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.304939    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_642803fa-d182-445e-a4da-f386cb75b3b0/alertmanager/0.log"
Jul 28 18:57:02.332126 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.332089    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_642803fa-d182-445e-a4da-f386cb75b3b0/config-reloader/0.log"
Jul 28 18:57:02.354292 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.354251    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_642803fa-d182-445e-a4da-f386cb75b3b0/kube-rbac-proxy-web/0.log"
Jul 28 18:57:02.374126 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.374103    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_642803fa-d182-445e-a4da-f386cb75b3b0/kube-rbac-proxy/0.log"
Jul 28 18:57:02.393926 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.393893    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_642803fa-d182-445e-a4da-f386cb75b3b0/kube-rbac-proxy-metric/0.log"
Jul 28 18:57:02.414661 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.414634    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_642803fa-d182-445e-a4da-f386cb75b3b0/prom-label-proxy/0.log"
Jul 28 18:57:02.444609 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.444587    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_642803fa-d182-445e-a4da-f386cb75b3b0/init-config-reloader/0.log"
Jul 28 18:57:02.519839 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.519799    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-xrstz_1087a8f2-182c-40ca-95f2-9f81ce0e4bec/cluster-monitoring-operator/0.log"
Jul 28 18:57:02.667564 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.667534    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-pq9fh_d59926dd-c34f-4486-bed5-853d8eb97e68/monitoring-plugin/0.log"
Jul 28 18:57:02.911287 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.911263    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvzl4_45737a07-b27b-4d1f-9603-93e426b9796b/node-exporter/0.log"
Jul 28 18:57:02.933280 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.933226    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvzl4_45737a07-b27b-4d1f-9603-93e426b9796b/kube-rbac-proxy/0.log"
Jul 28 18:57:02.965382 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:02.965359    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vvzl4_45737a07-b27b-4d1f-9603-93e426b9796b/init-textfile/0.log"
Jul 28 18:57:03.095623 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.095592    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73b2cd2f-d3ba-44e1-af6d-98af9c5548e4/prometheus/0.log"
Jul 28 18:57:03.113930 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.113896    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73b2cd2f-d3ba-44e1-af6d-98af9c5548e4/config-reloader/0.log"
Jul 28 18:57:03.138265 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.138242    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73b2cd2f-d3ba-44e1-af6d-98af9c5548e4/thanos-sidecar/0.log"
Jul 28 18:57:03.163819 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.163792    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73b2cd2f-d3ba-44e1-af6d-98af9c5548e4/kube-rbac-proxy-web/0.log"
Jul 28 18:57:03.184156 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.184064    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73b2cd2f-d3ba-44e1-af6d-98af9c5548e4/kube-rbac-proxy/0.log"
Jul 28 18:57:03.204223 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.204199    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73b2cd2f-d3ba-44e1-af6d-98af9c5548e4/kube-rbac-proxy-thanos/0.log"
Jul 28 18:57:03.223846 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.223816    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_73b2cd2f-d3ba-44e1-af6d-98af9c5548e4/init-config-reloader/0.log"
Jul 28 18:57:03.261166 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.261137    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-mhmsz_0be57644-399e-4f8e-9a93-f0d2105f4e8f/prometheus-operator/0.log"
Jul 28 18:57:03.279729 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.279705    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-mhmsz_0be57644-399e-4f8e-9a93-f0d2105f4e8f/kube-rbac-proxy/0.log"
Jul 28 18:57:03.431700 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.431666    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb788c5-nt92b_491df11b-d7b8-406e-a6ed-8a338b3c626c/thanos-query/0.log"
Jul 28 18:57:03.454677 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.454652    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb788c5-nt92b_491df11b-d7b8-406e-a6ed-8a338b3c626c/kube-rbac-proxy-web/0.log"
Jul 28 18:57:03.477297 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.477265    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb788c5-nt92b_491df11b-d7b8-406e-a6ed-8a338b3c626c/kube-rbac-proxy/0.log"
Jul 28 18:57:03.497400 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.497354    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb788c5-nt92b_491df11b-d7b8-406e-a6ed-8a338b3c626c/prom-label-proxy/0.log"
Jul 28 18:57:03.522406 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.522365    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb788c5-nt92b_491df11b-d7b8-406e-a6ed-8a338b3c626c/kube-rbac-proxy-rules/0.log"
Jul 28 18:57:03.544980 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:03.544955    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bb788c5-nt92b_491df11b-d7b8-406e-a6ed-8a338b3c626c/kube-rbac-proxy-metrics/0.log"
Jul 28 18:57:05.216570 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.216541    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/1.log"
Jul 28 18:57:05.225479 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.225450    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-2gspt_72263172-706a-41e4-89ab-b2c5a52d86ff/console-operator/2.log"
Jul 28 18:57:05.852156 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.852120    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"]
Jul 28 18:57:05.852518 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.852504    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" containerName="copy"
Jul 28 18:57:05.852564 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.852520    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" containerName="copy"
Jul 28 18:57:05.852564 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.852533    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" containerName="gather"
Jul 28 18:57:05.852564 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.852538    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" containerName="gather"
Jul 28 18:57:05.852660 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.852620    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" containerName="gather"
Jul 28 18:57:05.852660 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.852637    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e624ca03-3ad2-4320-8fa0-6d9dd8fd0435" containerName="copy"
Jul 28 18:57:05.855323 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.855295    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:05.857928 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.857905    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-67kr8\"/\"kube-root-ca.crt\""
Jul 28 18:57:05.858968 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.858939    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-67kr8\"/\"openshift-service-ca.crt\""
Jul 28 18:57:05.858968 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.858944    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-67kr8\"/\"default-dockercfg-5smz9\""
Jul 28 18:57:05.864289 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.864266    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"]
Jul 28 18:57:05.991261 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.991236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-podres\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:05.991408 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.991348    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-proc\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:05.991408 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.991377    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9d8m\" (UniqueName: \"kubernetes.io/projected/3c7191e6-a75c-4845-9bdb-457879897269-kube-api-access-r9d8m\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:05.991487 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.991414    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-sys\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:05.991487 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:05.991433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-lib-modules\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.092716 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.092681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-proc\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.092860 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.092722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9d8m\" (UniqueName: \"kubernetes.io/projected/3c7191e6-a75c-4845-9bdb-457879897269-kube-api-access-r9d8m\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.092860 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.092739    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-sys\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.092860 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.092755    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-lib-modules\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.092860 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.092790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-podres\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.092860 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.092808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-proc\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.092860 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.092831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-sys\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.093069 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.092914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-podres\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.093069 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.092914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c7191e6-a75c-4845-9bdb-457879897269-lib-modules\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.100560 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.100532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9d8m\" (UniqueName: \"kubernetes.io/projected/3c7191e6-a75c-4845-9bdb-457879897269-kube-api-access-r9d8m\") pod \"perf-node-gather-daemonset-4wgbp\" (UID: \"3c7191e6-a75c-4845-9bdb-457879897269\") " pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.165932 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.165869    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.292802 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.292754    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"]
Jul 28 18:57:06.297171 ip-10-0-136-111 kubenswrapper[2579]: W0728 18:57:06.297142    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3c7191e6_a75c_4845_9bdb_457879897269.slice/crio-c5d521b9d562cbf27f16bef63bde2c33d5041781fe5fcadb6668f1380403aace WatchSource:0}: Error finding container c5d521b9d562cbf27f16bef63bde2c33d5041781fe5fcadb6668f1380403aace: Status 404 returned error can't find the container with id c5d521b9d562cbf27f16bef63bde2c33d5041781fe5fcadb6668f1380403aace
Jul 28 18:57:06.342244 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.342220    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-ddr24_8cb83001-d5e0-4373-90bb-15a82271db50/volume-data-source-validator/0.log"
Jul 28 18:57:06.630553 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.630517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp" event={"ID":"3c7191e6-a75c-4845-9bdb-457879897269","Type":"ContainerStarted","Data":"aac05afbf14d07ab93e433204038b05ae25ea2f6068572270c9a3568722495cf"}
Jul 28 18:57:06.630757 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.630568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp" event={"ID":"3c7191e6-a75c-4845-9bdb-457879897269","Type":"ContainerStarted","Data":"c5d521b9d562cbf27f16bef63bde2c33d5041781fe5fcadb6668f1380403aace"}
Jul 28 18:57:06.630757 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.630663    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:06.646585 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:06.646541    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp" podStartSLOduration=1.6465261089999998 podStartE2EDuration="1.646526109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:57:06.644740029 +0000 UTC m=+6414.873939125" watchObservedRunningTime="2026-07-28 18:57:06.646526109 +0000 UTC m=+6414.875725204"
Jul 28 18:57:07.374571 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:07.374538    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-clksn_f1338c56-b7f9-4bec-b8a7-8dc14677efe3/dns/0.log"
Jul 28 18:57:07.394776 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:07.394753    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-clksn_f1338c56-b7f9-4bec-b8a7-8dc14677efe3/kube-rbac-proxy/0.log"
Jul 28 18:57:07.511639 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:07.511609    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jh65p_34d3b868-e31e-4fd6-9b00-f117a39ebbc3/dns-node-resolver/0.log"
Jul 28 18:57:08.059290 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:08.059259    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4w8n8_e925228f-d166-4239-8f86-dad787c3e6ee/node-ca/0.log"
Jul 28 18:57:09.477858 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:09.477830    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dc8cf7cff-gn2tw_e77175ad-f991-48f4-bfa2-b35bdbcb138e/router/0.log"
Jul 28 18:57:09.966008 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:09.965976    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-48qh4_a0767256-2cea-499e-8c75-568595e72e3d/serve-healthcheck-canary/0.log"
Jul 28 18:57:10.448608 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:10.448573    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-zxbld_38193ea1-7cb0-4c1e-9664-a9b02b706559/insights-operator/0.log"
Jul 28 18:57:10.452000 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:10.451972    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-zxbld_38193ea1-7cb0-4c1e-9664-a9b02b706559/insights-operator/1.log"
Jul 28 18:57:10.621571 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:10.621532    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fg6ls_b88bc623-3928-4437-9acc-a10ecec23114/kube-rbac-proxy/0.log"
Jul 28 18:57:10.640448 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:10.640419    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fg6ls_b88bc623-3928-4437-9acc-a10ecec23114/exporter/0.log"
Jul 28 18:57:10.661064 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:10.661034    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fg6ls_b88bc623-3928-4437-9acc-a10ecec23114/extractor/0.log"
Jul 28 18:57:12.645397 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:12.645370    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-67kr8/perf-node-gather-daemonset-4wgbp"
Jul 28 18:57:13.615116 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:13.615064    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-659b67c655-hmff7_5755e1cd-4dd5-428b-800d-92a8661b1b92/manager/0.log"
Jul 28 18:57:13.634741 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:13.634714    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-4tnc8_306ff766-a0c0-48bb-bc14-c972c472d504/openshift-lws-operator/0.log"
Jul 28 18:57:14.622007 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:14.621954    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-84fb8458f8-4rlxl_43911596-dc27-4427-b177-8e3d89f8cbf5/manager/0.log"
Jul 28 18:57:14.643049 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:14.643021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-w4pv6_a4d7e1a1-93ec-4800-acbe-117455e3e8e8/server/0.log"
Jul 28 18:57:16.064203 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:16.064174    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-zhv7r_1a7a30a2-26c8-4d15-89d8-c3fbc38858a0/manager/0.log"
Jul 28 18:57:16.510473 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:16.510443    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dc8cf7cff-gn2tw_e77175ad-f991-48f4-bfa2-b35bdbcb138e/router/0.log"
Jul 28 18:57:18.014463 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:18.014429    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5dc8cf7cff-gn2tw_e77175ad-f991-48f4-bfa2-b35bdbcb138e/router/0.log"
Jul 28 18:57:21.553636 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:21.553605    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n_e62fc7a6-3156-4e2d-a376-1a753ef8eb8d/kube-storage-version-migrator-operator/1.log"
Jul 28 18:57:21.555190 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:21.555164    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-7wl9n_e62fc7a6-3156-4e2d-a376-1a753ef8eb8d/kube-storage-version-migrator-operator/0.log"
Jul 28 18:57:22.551035 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:22.550961    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-57cvt_76de2b66-aaa2-426f-b965-89bd9d61a207/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:57:22.571927 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:22.571902    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-57cvt_76de2b66-aaa2-426f-b965-89bd9d61a207/egress-router-binary-copy/0.log"
Jul 28 18:57:22.592165 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:22.592142    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-57cvt_76de2b66-aaa2-426f-b965-89bd9d61a207/cni-plugins/0.log"
Jul 28 18:57:22.611889 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:22.611863    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-57cvt_76de2b66-aaa2-426f-b965-89bd9d61a207/bond-cni-plugin/0.log"
Jul 28 18:57:22.631161 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:22.631137    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-57cvt_76de2b66-aaa2-426f-b965-89bd9d61a207/routeoverride-cni/0.log"
Jul 28 18:57:22.649352 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:22.649323    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-57cvt_76de2b66-aaa2-426f-b965-89bd9d61a207/whereabouts-cni-bincopy/0.log"
Jul 28 18:57:22.668898 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:22.668877    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-57cvt_76de2b66-aaa2-426f-b965-89bd9d61a207/whereabouts-cni/0.log"
Jul 28 18:57:23.105873 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:23.105840    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-j4jsd_28c3098d-a6c1-4700-95fe-93e39e621f83/kube-multus/0.log"
Jul 28 18:57:23.193567 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:23.193533    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bx6mn_19846f2a-05eb-485f-8ae6-67e7090bc346/network-metrics-daemon/0.log"
Jul 28 18:57:23.209429 ip-10-0-136-111 kubenswrapper[2579]: I0728 18:57:23.209403    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bx6mn_19846f2a-05eb-485f-8ae6-67e7090bc346/kube-rbac-proxy/0.log"