Jul 09 09:33:25.888219 ip-10-0-129-157 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.280479    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287459    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287473    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287477    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287480    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287484    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287487    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287489    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:26.381720 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287500    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:26.355119 ip-10-0-129-157 systemd[1]: Started Kubernetes Kubelet.
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287503    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287506    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287509    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287512    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287516    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287518    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287521    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287524    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287527    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287529    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287532    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287535    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287537    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287540    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287542    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287545    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287549    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287553    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:26.382713 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287557    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287561    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287564    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287567    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287571    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287574    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287577    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287580    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287583    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287587    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287590    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287593    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287596    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287599    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287602    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287605    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287609    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287612    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287614    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287617    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:26.383974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287619    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287622    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287625    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287627    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287630    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287632    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287635    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287637    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287641    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287644    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287646    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287649    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287652    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287655    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287658    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287661    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287664    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287667    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287670    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287672    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:26.384642 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287675    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287677    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287680    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287682    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287685    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287687    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287690    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287694    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287697    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287700    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287703    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287705    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287708    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287710    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287713    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287716    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287718    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287721    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287723    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:26.385533 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287726    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287728    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287731    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287734    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287736    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287739    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287743    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287745    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287748    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287750    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287753    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287755    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287757    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.287760    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290738    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290749    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290752    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290757    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290761    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:26.386451 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290766    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290771    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290774    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290777    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290781    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290784    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290787    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290790    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290794    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290796    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290799    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290802    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290805    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290807    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290810    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290813    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290815    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290818    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290821    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:26.387260 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290825    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290829    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290831    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290834    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290837    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290844    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290847    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290850    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290853    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290856    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290859    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290862    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290865    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290867    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290870    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290873    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290876    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290878    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290881    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290883    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:26.388707 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290886    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290888    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290891    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290893    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290896    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290898    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290902    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290904    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290907    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290909    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290912    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290914    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290917    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290919    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290922    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290949    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290952    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290955    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290963    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290966    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:26.389483 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290969    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290972    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290975    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290978    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290981    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290983    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290986    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290989    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290991    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290993    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290996    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.290998    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291001    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291003    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291006    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291008    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291011    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291013    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291016    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291019    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:26.390257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291022    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291024    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291027    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291029    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291032    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291034    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291037    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291039    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291042    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291047    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291049    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291052    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291054    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291058    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291061    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291444    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291449    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291451    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291454    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291457    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:26.390735 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291459    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291461    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291464    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291467    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291470    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291472    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291475    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291477    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291480    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291483    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291485    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291488    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291491    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291494    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291496    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291499    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291501    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291504    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291506    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:26.391257 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291508    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291511    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291516    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291518    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291521    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291523    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291529    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291533    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291535    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291538    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291541    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291543    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291545    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291548    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291550    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291553    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291555    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291558    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291560    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291563    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:26.391716 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291565    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291568    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291570    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291572    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291575    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291578    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291582    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291584    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291587    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291589    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291593    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291596    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291599    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291602    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291606    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291609    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291612    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291614    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291619    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:26.392238 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291622    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291626    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291628    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291631    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291634    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291637    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291639    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291643    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291645    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291648    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291650    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291653    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291655    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291658    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291660    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291663    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291665    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291668    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291670    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291674    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:26.392688 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291676    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291678    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291681    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291684    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291686    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291689    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291691    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291695    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291697    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291700    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291704    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291708    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291710    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291713    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291715    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.291717    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291792    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291802    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291809    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291816    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291822    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 09:33:26.393181 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291827    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291833    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291838    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291841    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291844    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291847    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291850    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291853    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291856    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291859    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291862    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291866    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291869    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291874    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291877    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291880    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291883    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291886    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291891    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291894    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291897    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291900    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291904    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291907    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 09:33:26.393677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291910    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291913    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291916    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291920    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291923    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291940    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291944    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291947    2581 flags.go:64] FLAG: --enable-server="true"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291950    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291954    2581 flags.go:64] FLAG: --event-burst="100"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291956    2581 flags.go:64] FLAG: --event-qps="50"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291959    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291962    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291965    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291968    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291971    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291974    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291977    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291980    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291983    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291986    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291989    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291992    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291994    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.291997    2581 flags.go:64] FLAG: --feature-gates=""
Jul 09 09:33:26.394257 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292001    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292005    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292008    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292011    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292014    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292019    2581 flags.go:64] FLAG: --help="false"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292022    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-129-157.ec2.internal"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292025    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292028    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292030    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292034    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292037    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292040    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292043    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292045    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292048    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292051    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292054    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292057    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292059    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292062    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292065    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292068    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292070    2581 flags.go:64] FLAG: --lock-file=""
Jul 09 09:33:26.394901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292073    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292076    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292079    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292085    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292088    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292091    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292094    2581 flags.go:64] FLAG: --logging-format="text"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292096    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292100    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292104    2581 flags.go:64] FLAG: --manifest-url=""
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292107    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292111    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292114    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292120    2581 flags.go:64] FLAG: --max-pods="110"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292123    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292126    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292129    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292132    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292135    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292138    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292141    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292148    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292151    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292154    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 09:33:26.395500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292157    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292160    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292166    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292169    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292172    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292175    2581 flags.go:64] FLAG: --port="10250"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292178    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292181    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00e34a7456fc2d521"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292184    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292187    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292190    2581 flags.go:64] FLAG: --register-node="true"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292192    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292196    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292199    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292202    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292205    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292209    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292212    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292216    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292219    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292221    2581 flags.go:64] FLAG: --runonce="false"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292226    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292228    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292231    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292234    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 09:33:26.396150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292237    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292240    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292243    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292246    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292248    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292251    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292254    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292257    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292260    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292263    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292266    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292271    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292274    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292276    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292281    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292283    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292286    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292289    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292292    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292295    2581 flags.go:64] FLAG: --v="2"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292298    2581 flags.go:64] FLAG: --version="false"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292306    2581 flags.go:64] FLAG: --vmodule=""
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292310    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.292313    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292429    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:26.396848 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292434    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292437    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292440    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292444    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292447    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292449    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292452    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292454    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292458    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292462    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292465    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292468    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292471    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292474    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292477    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292480    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292482    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292485    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292488    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292490    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:26.397603 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292493    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292495    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292498    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292500    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292503    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292506    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292508    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292510    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292513    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292516    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292518    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292521    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292525    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292527    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292530    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292534    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292536    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292539    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292541    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292544    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:26.398183 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292546    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292548    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292551    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292553    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292556    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292558    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292561    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292563    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292566    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292569    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292571    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292574    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292576    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292579    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292581    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292584    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292586    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292589    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292591    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292594    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:26.398749 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292596    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292598    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292602    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292605    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292608    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292611    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292613    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292617    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292620    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292623    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292625    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292628    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292630    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292633    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292635    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292638    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292640    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292643    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292645    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292648    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:26.399313 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292650    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292653    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292656    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292658    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292661    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292663    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292666    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292668    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292672    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292675    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292677    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292680    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292683    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292685    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292688    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292690    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292695    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:26.400156 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.292699    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.293393    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.299914    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.299942    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.299962    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.299968    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300030    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300035    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300039    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300042    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300044    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300047    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300050    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300052    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300055    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:26.400625 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300058    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300060    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300063    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300065    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300068    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300071    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300074    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300077    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300080    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300084    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300087    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300091    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300094    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300097    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300100    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300103    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300105    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300108    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300111    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:26.401020 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300113    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300117    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300119    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300122    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300126    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300129    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300132    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300135    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300138    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300141    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300144    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300147    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300150    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300152    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300155    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300158    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300160    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300163    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300166    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:26.401482 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300169    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300171    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300174    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300177    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300179    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300182    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300185    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300188    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300190    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300193    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300196    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300198    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300201    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300203    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300206    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300209    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300212    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300214    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300217    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300220    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:26.401944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300222    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300225    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300231    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300233    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300236    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300238    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300241    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300243    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300246    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300248    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300251    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300254    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300256    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300259    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300262    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300264    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300267    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300270    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300273    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300275    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:26.402464 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300278    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300281    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300284    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300286    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300289    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300293    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300297    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300300    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300302    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300305    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300308    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300310    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.300315    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300449    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300454    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 09:33:26.402944 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300459    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300461    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300464    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300467    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300470    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300473    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300477    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300480    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300483    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300486    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300488    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300491    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300494    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300496    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300499    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300501    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300504    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300506    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300509    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300512    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 09:33:26.403308 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300514    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300517    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300519    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300523    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300525    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300528    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300531    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300533    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300536    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300539    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300541    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300543    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300547    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300550    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300552    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300555    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300558    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300561    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300565    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300568    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 09:33:26.403793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300570    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300573    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300575    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300578    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300581    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300583    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300586    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300588    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300591    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300594    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300596    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300599    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300601    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300604    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300607    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300610    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300612    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300615    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300617    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300626    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 09:33:26.404365 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300628    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300631    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300633    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300637    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300641    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300645    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300648    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300651    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300654    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300657    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300661    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300663    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300666    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300668    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300671    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300673    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300676    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300678    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300681    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 09:33:26.404844 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300684    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300686    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300689    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300692    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300694    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300697    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300700    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300702    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300705    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300708    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300712    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300714    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300717    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300728    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300730    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300733    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300736    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 09:33:26.405316 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:26.300740    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.300745    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.300867    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.302911    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.304370    2581 server.go:1019] "Starting client certificate rotation"
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.304467    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.304499    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.307056    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.308449    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.320963    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.322915    2581 log.go:25] "Validated CRI v1 image API"
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.324315    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 09:33:26.405722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.327889    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ca9822d-be84-45c9-9952-4de7486c1a4a:/dev/nvme0n1p4 9a18c769-243b-452d-98b1-65880d1dd268:/dev/nvme0n1p3]
Jul 09 09:33:26.406045 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.327902    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 09:33:26.406045 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.333793    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.333940    2581 manager.go:216] Machine: {Timestamp:2026-07-09 09:33:26.33261446 +0000 UTC m=+0.323537312 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098725 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec293af8fba70d18a0902fd8a8277dbc SystemUUID:ec293af8-fba7-0d18-a090-2fd8a8277dbc BootID:5a7de270-3337-4ec9-a0c9-39bc2db32ca3 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3f:c2:73:dc:ef Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3f:c2:73:dc:ef Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3e:58:bf:da:4e:b5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.334078    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.334187    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.337613    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.337631    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-157.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.339014    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.339025    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.339032    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.339675    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.341012    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.343693    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.343798    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.343820    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.343844    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.343858    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.343867    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.343876    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 09:33:26.406103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.344174    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.344189    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.346149    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.347906    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.347951    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349005    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349020    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349027    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349036    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349047    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349056    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349068    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349075    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349083    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349090    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349114    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.349122    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.350498    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.350505    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.353543    2581 server.go:1268] "Started kubelet"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.353721    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r6q64"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.353668    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.353813    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.354118    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:26.357470    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.357726    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.358010    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-157.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.359591    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:26.359694    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.361903    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r6q64"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:26.363651    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:26.361832    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-157.ec2.internal.18c0954c0fc7c117  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-157.ec2.internal,UID:ip-10-0-129-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-157.ec2.internal,},FirstTimestamp:2026-07-09 09:33:26.353518871 +0000 UTC m=+0.344441717,LastTimestamp:2026-07-09 09:33:26.353518871 +0000 UTC m=+0.344441717,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-157.ec2.internal,}"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.368068    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.368078    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:26.368139    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.368617    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.368650    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.368615    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.368708    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.368718    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 09:33:26.406577 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.369020    2581 factory.go:55] Registering systemd factory
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.369504    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.369712    2581 factory.go:153] Registering CRI-O factory
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.369724    2581 factory.go:223] Registration of the crio container factory successfully
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.369771    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.369797    2581 factory.go:103] Registering Raw factory
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.369816    2581 manager.go:1196] Started watching for new ooms in manager
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.370282    2581 manager.go:349] Starting recovery of all containers
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.370746    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:26.376187    2581 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-157.ec2.internal\" not found" node="ip-10-0-129-157.ec2.internal"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.379669    2581 manager.go:354] Recovery completed
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.385567    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.388322    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.388354    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.388366    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.388915    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.388941    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.388955    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.388961    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.389785    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.389824    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.389841    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.390410    2581 policy_none.go:49] "None policy: Start"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.390424    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.390434    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 09:33:26.407559 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.395325    2581 policy_none.go:47] "Start"
Jul 09 09:33:26.464796 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.464769    2581 manager.go:337] "Starting Device Plugin manager"
Jul 09 09:33:26.464916 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:26.464832    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 09:33:26.464916 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.464848    2581 server.go:87] "Starting device plugin registration server"
Jul 09 09:33:26.465229 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.465210    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 09:33:26.465341 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.465230    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 09:33:26.466097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.465477    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 09:33:26.466097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.465736    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 09:33:26.466097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:26.465744    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 09:33:26.466097 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:26.466081    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 09:33:26.466275 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:26.466154    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:28.221145 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.221107    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 09:33:28.222656 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.222314    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 09:33:28.222656 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.222358    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 09:33:28.222656 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.222422    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 09:33:28.222656 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.222462    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 09:33:28.225795 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.225776    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:28.232101 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.232086    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:28.233136 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.233119    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:28.233232 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.233154    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:28.233232 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.233170    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:28.233232 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.233200    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.245598 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.245583    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.245675 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.245603    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-157.ec2.internal\": node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:28.270918 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.270880    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:28.323467 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.323364    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal"]
Jul 09 09:33:28.323537 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.323512    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:28.324432 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.324416    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:28.324508 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.324443    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:28.324508 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.324454    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:28.325603 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.325591    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:28.325743 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.325729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.325783 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.325758    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:28.326510 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.326495    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:28.326591 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.326496    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:28.326591 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.326535    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:28.326591 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.326545    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:28.326591 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.326517    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:28.326768 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.326615    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:28.327642 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.327629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.327694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.327652    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 09:33:28.328337 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.328323    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 09:33:28.328394 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.328342    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 09:33:28.328394 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.328356    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeHasSufficientPID"
Jul 09 09:33:28.336116 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.336100    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2c0eff54d21d6540cf23785ec244815-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal\" (UID: \"d2c0eff54d21d6540cf23785ec244815\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.336195 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.336127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2c0eff54d21d6540cf23785ec244815-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal\" (UID: \"d2c0eff54d21d6540cf23785ec244815\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.336195 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.336149    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/78255babc9a1b932c1400b78f8b89518-config\") pod \"kube-apiserver-proxy-ip-10-0-129-157.ec2.internal\" (UID: \"78255babc9a1b932c1400b78f8b89518\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.350153 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.350108    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-157.ec2.internal\" not found" node="ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.354056 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.354040    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-157.ec2.internal\" not found" node="ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.371927 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.371910    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:28.436386 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.436359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2c0eff54d21d6540cf23785ec244815-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal\" (UID: \"d2c0eff54d21d6540cf23785ec244815\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.436504 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.436390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2c0eff54d21d6540cf23785ec244815-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal\" (UID: \"d2c0eff54d21d6540cf23785ec244815\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.436504 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.436407    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/78255babc9a1b932c1400b78f8b89518-config\") pod \"kube-apiserver-proxy-ip-10-0-129-157.ec2.internal\" (UID: \"78255babc9a1b932c1400b78f8b89518\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.436504 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.436448    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/78255babc9a1b932c1400b78f8b89518-config\") pod \"kube-apiserver-proxy-ip-10-0-129-157.ec2.internal\" (UID: \"78255babc9a1b932c1400b78f8b89518\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.436504 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.436465    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2c0eff54d21d6540cf23785ec244815-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal\" (UID: \"d2c0eff54d21d6540cf23785ec244815\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.436504 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.436462    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2c0eff54d21d6540cf23785ec244815-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal\" (UID: \"d2c0eff54d21d6540cf23785ec244815\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.472810 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.472780    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:28.573254 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.573213    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:28.653783 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.653724    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.657872 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.657671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:28.673532 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.673510    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:28.774047 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.773999    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:28.874458 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.874434    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:28.911039 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.910973    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:28.970090 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.970062    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 09:33:28.970216 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.970198    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 09:33:28.970255 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.970236    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 09:33:28.970288 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:28.970242    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 09:33:28.975366 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:28.975348    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:29.031502 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:29.031460    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 09:28:28 +0000 UTC" deadline="2028-04-18 22:52:24.611548724 +0000 UTC"
Jul 09 09:33:29.031502 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:29.031495    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15589h18m55.58005736s"
Jul 09 09:33:29.076291 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:29.076264    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:29.177022 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:29.176940    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-157.ec2.internal\" not found"
Jul 09 09:33:29.240985 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:29.240949    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c0eff54d21d6540cf23785ec244815.slice/crio-a1a1028d7c97ce525a41a65127a3a2103102612558e84b459ea7ff9ed9bd0e2d WatchSource:0}: Error finding container a1a1028d7c97ce525a41a65127a3a2103102612558e84b459ea7ff9ed9bd0e2d: Status 404 returned error can't find the container with id a1a1028d7c97ce525a41a65127a3a2103102612558e84b459ea7ff9ed9bd0e2d
Jul 09 09:33:29.241354 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:29.241341    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78255babc9a1b932c1400b78f8b89518.slice/crio-698ab13a06b5a2ca9476911bb0e372588894404429e1dc7fec75654b1c6edfae WatchSource:0}: Error finding container 698ab13a06b5a2ca9476911bb0e372588894404429e1dc7fec75654b1c6edfae: Status 404 returned error can't find the container with id 698ab13a06b5a2ca9476911bb0e372588894404429e1dc7fec75654b1c6edfae
Jul 09 09:33:29.245356 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:29.245341    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 09:33:29.248023 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:29.248009    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:29.334649 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:29.334614    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:29.346767 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:29.346745    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 09:33:29.347729 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:29.347716    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal"
Jul 09 09:33:29.357362 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:29.357348    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 09:33:30.011904 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.011867    2581 apiserver.go:52] "Watching apiserver"
Jul 09 09:33:30.019989 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.019968    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 09:33:30.020409 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.020386    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-fxrwr","openshift-multus/network-metrics-daemon-5g56w","openshift-network-operator/iptables-alerter-2r6dl","kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal","openshift-cluster-node-tuning-operator/tuned-kwccx","openshift-image-registry/node-ca-2fd5t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal","openshift-multus/multus-additional-cni-plugins-2tnhw","openshift-network-diagnostics/network-check-target-9gj6d","openshift-ovn-kubernetes/ovnkube-node-vmx6b","kube-system/global-pull-secret-syncer-wg7tt","kube-system/konnectivity-agent-x6tpp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq","openshift-dns/node-resolver-pk5f2"]
Jul 09 09:33:30.022095 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.022072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.023322 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.023297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:30.023411 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.023365    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:30.024557 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.024384    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 09:33:30.024557 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.024418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 09:33:30.024557 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.024451    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8b6h9\""
Jul 09 09:33:30.025674 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.024639    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 09:33:30.025674 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.025549    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.027380 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.027072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.028775 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.028369    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 09:33:30.028775 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.028404    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 09:33:30.028933 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.028807    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fn48m\""
Jul 09 09:33:30.028933 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.028917    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.029036 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.028973    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kms9d\""
Jul 09 09:33:30.030967 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.029988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.030967 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.030524    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 09:33:30.030967 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.030769    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 09:33:30.030967 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.030798    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 09:33:30.031193 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.031080    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 09:33:30.031992 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.031478    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 09:33:30.031992 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.031731    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 09:33:30.031992 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.031951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.033440 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.033077    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mglgr\""
Jul 09 09:33:30.033440 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.033346    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 09:33:30.033570 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.033447    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 09:33:30.033734 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.033705    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9vhdz\""
Jul 09 09:33:30.034167 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.034127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.034464 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.034447    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 09:33:30.035124 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.035103    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 09:33:30.036193 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.035404    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:30.036193 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.035463    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:30.036193 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.035504    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 09:33:30.036193 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.035732    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 09:33:30.036193 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.035967    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xxt2g\""
Jul 09 09:33:30.036193 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.036052    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 09:33:30.036583 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.036394    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 09:33:30.036583 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.036414    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 09:33:30.037065 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.036970    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.037475 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.037448    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wq74z\""
Jul 09 09:33:30.038524 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.038429    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:30.038524 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.038489    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:30.038923 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.038904    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 09:33:30.039288 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.039268    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 09:33:30.039436 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.039419    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 09:33:30.039588 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.039571    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cshn4\""
Jul 09 09:33:30.039674 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.039657    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 09:33:30.040234 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.040062    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 09:33:30.040234 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.040150    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:30.040234 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.040158    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 09:33:30.042479 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.042450    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k8c2g\""
Jul 09 09:33:30.042642 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.042506    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 09:33:30.042708 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.042689    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 09:33:30.045728 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.045705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.045833 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.045742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-var-lib-kubelet\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.045833 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.045767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-host\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.045833 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.045812    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-tuned\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.045993 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.045846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-socket-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.045993 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.045875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-registration-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.045993 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.045932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-sys-fs\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.045993 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.045956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-cni-dir\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.046189 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-systemd\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.046189 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh6zw\" (UniqueName: \"kubernetes.io/projected/f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4-kube-api-access-gh6zw\") pod \"node-resolver-pk5f2\" (UID: \"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4\") " pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.046189 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-system-cni-dir\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.046189 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046127    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-etc-selinux\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.046189 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046150    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-var-lib-cni-bin\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.046417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-os-release\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.046417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-cnibin\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.046417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-run-k8s-cni-cncf-io\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.046417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-var-lib-cni-multus\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.046417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj8bt\" (UniqueName: \"kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt\") pod \"network-check-target-9gj6d\" (UID: \"1e83b238-3f02-48c8-a05a-045a2e0fc781\") " pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:30.046417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-sys\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.046417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046412    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 09:33:30.046417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-cnibin\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.046754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.046754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:30.046754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-run\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.046754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8eb0783e-86d2-42bb-96e8-c69c1fae5629-host\") pod \"node-ca-2fd5t\" (UID: \"8eb0783e-86d2-42bb-96e8-c69c1fae5629\") " pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.046754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jgq4\" (UniqueName: \"kubernetes.io/projected/1394ed00-564d-4d9d-87f1-8990a5f44301-kube-api-access-6jgq4\") pod \"iptables-alerter-2r6dl\" (UID: \"1394ed00-564d-4d9d-87f1-8990a5f44301\") " pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.046754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046597    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-device-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.046754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-socket-dir-parent\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.046754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-system-cni-dir\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.046754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-modprobe-d\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.046754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-sysconfig\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21a09ec5-5646-45b9-bcce-9c96d2e9537c-cni-binary-copy\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-var-lib-kubelet\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046816    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-kubernetes\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxtbd\" (UniqueName: \"kubernetes.io/projected/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-kube-api-access-rxtbd\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046865    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-hostroot\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6dt4\" (UniqueName: \"kubernetes.io/projected/21a09ec5-5646-45b9-bcce-9c96d2e9537c-kube-api-access-s6dt4\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-os-release\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.046992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8eb0783e-86d2-42bb-96e8-c69c1fae5629-serviceca\") pod \"node-ca-2fd5t\" (UID: \"8eb0783e-86d2-42bb-96e8-c69c1fae5629\") " pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4-hosts-file\") pod \"node-resolver-pk5f2\" (UID: \"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4\") " pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.047088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-run-multus-certs\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-etc-kubernetes\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srftw\" (UniqueName: \"kubernetes.io/projected/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-kube-api-access-srftw\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047157    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfp6g\" (UniqueName: \"kubernetes.io/projected/cc58490a-3112-469d-b967-06c5bd8c9e20-kube-api-access-lfp6g\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-tmp\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1394ed00-564d-4d9d-87f1-8990a5f44301-host-slash\") pod \"iptables-alerter-2r6dl\" (UID: \"1394ed00-564d-4d9d-87f1-8990a5f44301\") " pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-run-netns\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047255    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxw4s\" (UniqueName: \"kubernetes.io/projected/8eb0783e-86d2-42bb-96e8-c69c1fae5629-kube-api-access-wxw4s\") pod \"node-ca-2fd5t\" (UID: \"8eb0783e-86d2-42bb-96e8-c69c1fae5629\") " pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpm92\" (UniqueName: \"kubernetes.io/projected/2f7386da-1749-4189-ab62-665534f4344b-kube-api-access-mpm92\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1394ed00-564d-4d9d-87f1-8990a5f44301-iptables-alerter-script\") pod \"iptables-alerter-2r6dl\" (UID: \"1394ed00-564d-4d9d-87f1-8990a5f44301\") " pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-conf-dir\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-sysctl-d\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-sysctl-conf\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047443    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-lib-modules\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-daemon-config\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.047523 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.047497    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4-tmp-dir\") pod \"node-resolver-pk5f2\" (UID: \"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4\") " pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.068298 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.068273    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wbrzz"
Jul 09 09:33:30.076605 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.076588    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wbrzz"
Jul 09 09:33:30.135353 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.135325    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 09:33:30.147644 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-run\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.147644 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8eb0783e-86d2-42bb-96e8-c69c1fae5629-host\") pod \"node-ca-2fd5t\" (UID: \"8eb0783e-86d2-42bb-96e8-c69c1fae5629\") " pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.147844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-var-lib-openvswitch\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.147844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147712    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.147844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jgq4\" (UniqueName: \"kubernetes.io/projected/1394ed00-564d-4d9d-87f1-8990a5f44301-kube-api-access-6jgq4\") pod \"iptables-alerter-2r6dl\" (UID: \"1394ed00-564d-4d9d-87f1-8990a5f44301\") " pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.147844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-device-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.147844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147779    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8eb0783e-86d2-42bb-96e8-c69c1fae5629-host\") pod \"node-ca-2fd5t\" (UID: \"8eb0783e-86d2-42bb-96e8-c69c1fae5629\") " pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.147844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147796    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-kubelet-config\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:30.147844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147799    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-run\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.147844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-socket-dir-parent\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-system-cni-dir\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147902    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-system-cni-dir\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-modprobe-d\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-sysconfig\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147978    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-socket-dir-parent\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.147990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-kubelet\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148038    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-device-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148040    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21a09ec5-5646-45b9-bcce-9c96d2e9537c-cni-binary-copy\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148088    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-sysconfig\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148096    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-modprobe-d\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-var-lib-kubelet\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-kubernetes\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148243    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-var-lib-kubelet\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-dbus\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxtbd\" (UniqueName: \"kubernetes.io/projected/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-kube-api-access-rxtbd\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.148345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148350    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-kubernetes\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148380    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-log-socket\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148409    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-cni-bin\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9e611ab-51d8-4f28-9106-829c3202216c-ovnkube-config\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-hostroot\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6dt4\" (UniqueName: \"kubernetes.io/projected/21a09ec5-5646-45b9-bcce-9c96d2e9537c-kube-api-access-s6dt4\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148535    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148583    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-hostroot\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-os-release\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148696    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/21a09ec5-5646-45b9-bcce-9c96d2e9537c-cni-binary-copy\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148705    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-os-release\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8eb0783e-86d2-42bb-96e8-c69c1fae5629-serviceca\") pod \"node-ca-2fd5t\" (UID: \"8eb0783e-86d2-42bb-96e8-c69c1fae5629\") " pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4-hosts-file\") pod \"node-resolver-pk5f2\" (UID: \"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4\") " pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-slash\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-run-multus-certs\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.149183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148804    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4-hosts-file\") pod \"node-resolver-pk5f2\" (UID: \"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4\") " pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148817    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-etc-kubernetes\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148839    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-run-multus-certs\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srftw\" (UniqueName: \"kubernetes.io/projected/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-kube-api-access-srftw\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148866    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-etc-kubernetes\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfp6g\" (UniqueName: \"kubernetes.io/projected/cc58490a-3112-469d-b967-06c5bd8c9e20-kube-api-access-lfp6g\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148909    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-tmp\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4efaeb6b-b220-4e84-ac19-115ce6ef37e8-konnectivity-ca\") pod \"konnectivity-agent-x6tpp\" (UID: \"4efaeb6b-b220-4e84-ac19-115ce6ef37e8\") " pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.148977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1394ed00-564d-4d9d-87f1-8990a5f44301-host-slash\") pod \"iptables-alerter-2r6dl\" (UID: \"1394ed00-564d-4d9d-87f1-8990a5f44301\") " pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-run-netns\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxw4s\" (UniqueName: \"kubernetes.io/projected/8eb0783e-86d2-42bb-96e8-c69c1fae5629-kube-api-access-wxw4s\") pod \"node-ca-2fd5t\" (UID: \"8eb0783e-86d2-42bb-96e8-c69c1fae5629\") " pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-systemd-units\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149076    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1394ed00-564d-4d9d-87f1-8990a5f44301-host-slash\") pod \"iptables-alerter-2r6dl\" (UID: \"1394ed00-564d-4d9d-87f1-8990a5f44301\") " pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149088    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9e611ab-51d8-4f28-9106-829c3202216c-ovnkube-script-lib\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149141    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-run-netns\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpm92\" (UniqueName: \"kubernetes.io/projected/2f7386da-1749-4189-ab62-665534f4344b-kube-api-access-mpm92\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.150061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149172    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8eb0783e-86d2-42bb-96e8-c69c1fae5629-serviceca\") pod \"node-ca-2fd5t\" (UID: \"8eb0783e-86d2-42bb-96e8-c69c1fae5629\") " pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1394ed00-564d-4d9d-87f1-8990a5f44301-iptables-alerter-script\") pod \"iptables-alerter-2r6dl\" (UID: \"1394ed00-564d-4d9d-87f1-8990a5f44301\") " pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-conf-dir\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-sysctl-d\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149261    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-cni-binary-copy\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-sysctl-conf\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149280    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-conf-dir\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149276    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-lib-modules\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-run-openvswitch\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-daemon-config\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4-tmp-dir\") pod \"node-resolver-pk5f2\" (UID: \"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4\") " pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149388    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-sysctl-d\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-run-ovn\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-cni-netd\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149442    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-sysctl-conf\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4efaeb6b-b220-4e84-ac19-115ce6ef37e8-agent-certs\") pod \"konnectivity-agent-x6tpp\" (UID: \"4efaeb6b-b220-4e84-ac19-115ce6ef37e8\") " pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:30.150819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149496    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-lib-modules\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-var-lib-kubelet\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-host\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-tuned\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c47q5\" (UniqueName: \"kubernetes.io/projected/c9e611ab-51d8-4f28-9106-829c3202216c-kube-api-access-c47q5\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149700    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1394ed00-564d-4d9d-87f1-8990a5f44301-iptables-alerter-script\") pod \"iptables-alerter-2r6dl\" (UID: \"1394ed00-564d-4d9d-87f1-8990a5f44301\") " pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-socket-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149734    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149766    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4-tmp-dir\") pod \"node-resolver-pk5f2\" (UID: \"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4\") " pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149777    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-host\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-registration-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-sys-fs\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-etc-openvswitch\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149863    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-registration-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-cni-dir\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-systemd\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.151478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149937    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-sys-fs\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh6zw\" (UniqueName: \"kubernetes.io/projected/f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4-kube-api-access-gh6zw\") pod \"node-resolver-pk5f2\" (UID: \"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4\") " pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149937    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-daemon-config\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.149967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-run-netns\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150054    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-socket-dir\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150053    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-multus-cni-dir\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150078    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-node-log\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150141    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-systemd\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-system-cni-dir\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-etc-selinux\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150244    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-system-cni-dir\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150253    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-var-lib-cni-bin\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-os-release\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-cnibin\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150334    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-var-lib-cni-bin\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150368    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-cnibin\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.152086 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150370    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-var-lib-kubelet\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150303    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f7386da-1749-4189-ab62-665534f4344b-etc-selinux\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9e611ab-51d8-4f28-9106-829c3202216c-ovn-node-metrics-cert\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150410    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-os-release\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-run-k8s-cni-cncf-io\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-var-lib-cni-multus\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8bt\" (UniqueName: \"kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt\") pod \"network-check-target-9gj6d\" (UID: \"1e83b238-3f02-48c8-a05a-045a2e0fc781\") " pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150531    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-var-lib-cni-multus\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150531    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-host-run-k8s-cni-cncf-io\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-sys\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-run-systemd\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9e611ab-51d8-4f28-9106-829c3202216c-env-overrides\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-cnibin\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150643    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-sys\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150660    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150693    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/21a09ec5-5646-45b9-bcce-9c96d2e9537c-cnibin\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.152694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:30.153360 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.150824    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.153360 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.150846    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:30.153360 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.150945    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs podName:cc58490a-3112-469d-b967-06c5bd8c9e20 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:30.650920171 +0000 UTC m=+2.975882451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs") pod "network-metrics-daemon-5g56w" (UID: "cc58490a-3112-469d-b967-06c5bd8c9e20") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:30.153360 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.153005    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-tmp\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.153360 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.153061    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-etc-tuned\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.160246 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.160219    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxtbd\" (UniqueName: \"kubernetes.io/projected/442a1957-0ac5-4abb-a1ea-cc11181c6e5a-kube-api-access-rxtbd\") pod \"tuned-kwccx\" (UID: \"442a1957-0ac5-4abb-a1ea-cc11181c6e5a\") " pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.160679 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.160653    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jgq4\" (UniqueName: \"kubernetes.io/projected/1394ed00-564d-4d9d-87f1-8990a5f44301-kube-api-access-6jgq4\") pod \"iptables-alerter-2r6dl\" (UID: \"1394ed00-564d-4d9d-87f1-8990a5f44301\") " pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.161461 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.161424    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srftw\" (UniqueName: \"kubernetes.io/projected/7320fb87-f8d7-4d0c-9e3a-5da86b108b62-kube-api-access-srftw\") pod \"multus-additional-cni-plugins-2tnhw\" (UID: \"7320fb87-f8d7-4d0c-9e3a-5da86b108b62\") " pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.161675 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.161654    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpm92\" (UniqueName: \"kubernetes.io/projected/2f7386da-1749-4189-ab62-665534f4344b-kube-api-access-mpm92\") pod \"aws-ebs-csi-driver-node-kn9kq\" (UID: \"2f7386da-1749-4189-ab62-665534f4344b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.161786 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.161766    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh6zw\" (UniqueName: \"kubernetes.io/projected/f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4-kube-api-access-gh6zw\") pod \"node-resolver-pk5f2\" (UID: \"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4\") " pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.162263 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.162022    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:30.162263 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.162047    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:30.162263 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.162059    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sj8bt for pod openshift-network-diagnostics/network-check-target-9gj6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:30.162263 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.162124    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt podName:1e83b238-3f02-48c8-a05a-045a2e0fc781 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:30.662108355 +0000 UTC m=+2.987070621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sj8bt" (UniqueName: "kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt") pod "network-check-target-9gj6d" (UID: "1e83b238-3f02-48c8-a05a-045a2e0fc781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:30.162591 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.162566    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfp6g\" (UniqueName: \"kubernetes.io/projected/cc58490a-3112-469d-b967-06c5bd8c9e20-kube-api-access-lfp6g\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:30.164756 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.164731    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxw4s\" (UniqueName: \"kubernetes.io/projected/8eb0783e-86d2-42bb-96e8-c69c1fae5629-kube-api-access-wxw4s\") pod \"node-ca-2fd5t\" (UID: \"8eb0783e-86d2-42bb-96e8-c69c1fae5629\") " pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.164924 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.164907    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6dt4\" (UniqueName: \"kubernetes.io/projected/21a09ec5-5646-45b9-bcce-9c96d2e9537c-kube-api-access-s6dt4\") pod \"multus-fxrwr\" (UID: \"21a09ec5-5646-45b9-bcce-9c96d2e9537c\") " pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.229317 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.229264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal" event={"ID":"d2c0eff54d21d6540cf23785ec244815","Type":"ContainerStarted","Data":"a1a1028d7c97ce525a41a65127a3a2103102612558e84b459ea7ff9ed9bd0e2d"}
Jul 09 09:33:30.230390 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.230364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal" event={"ID":"78255babc9a1b932c1400b78f8b89518","Type":"ContainerStarted","Data":"698ab13a06b5a2ca9476911bb0e372588894404429e1dc7fec75654b1c6edfae"}
Jul 09 09:33:30.251266 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-run-netns\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251266 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-node-log\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251469 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251300    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9e611ab-51d8-4f28-9106-829c3202216c-ovn-node-metrics-cert\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251469 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:30.251469 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-run-systemd\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251469 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9e611ab-51d8-4f28-9106-829c3202216c-env-overrides\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251469 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251367    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-run-netns\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251469 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251376    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-node-log\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251469 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251411    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-run-systemd\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.251484    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-var-lib-openvswitch\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-kubelet-config\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251599    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-var-lib-openvswitch\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.251613    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret podName:8aa5ea73-c624-40b0-8672-e1b09f0ee72b nodeName:}" failed. No retries permitted until 2026-07-09 09:33:30.751587218 +0000 UTC m=+3.076549491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret") pod "global-pull-secret-syncer-wg7tt" (UID: "8aa5ea73-c624-40b0-8672-e1b09f0ee72b") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251626    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-kubelet-config\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251618    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-run-ovn-kubernetes\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-kubelet\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-dbus\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-log-socket\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-cni-bin\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.251789 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251785    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9e611ab-51d8-4f28-9106-829c3202216c-ovnkube-config\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251808    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-dbus\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-slash\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251830    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-kubelet\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4efaeb6b-b220-4e84-ac19-115ce6ef37e8-konnectivity-ca\") pod \"konnectivity-agent-x6tpp\" (UID: \"4efaeb6b-b220-4e84-ac19-115ce6ef37e8\") " pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251862    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-cni-bin\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-systemd-units\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251909    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-log-socket\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251926    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c9e611ab-51d8-4f28-9106-829c3202216c-env-overrides\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251954    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-slash\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9e611ab-51d8-4f28-9106-829c3202216c-ovnkube-script-lib\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.251988    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-systemd-units\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252003    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-run-openvswitch\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-run-ovn\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-cni-netd\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4efaeb6b-b220-4e84-ac19-115ce6ef37e8-agent-certs\") pod \"konnectivity-agent-x6tpp\" (UID: \"4efaeb6b-b220-4e84-ac19-115ce6ef37e8\") " pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:30.252424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47q5\" (UniqueName: \"kubernetes.io/projected/c9e611ab-51d8-4f28-9106-829c3202216c-kube-api-access-c47q5\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.253184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252116    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-run-openvswitch\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.253184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252002    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.253184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-etc-openvswitch\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.253184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252151    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-run-ovn\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.253184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252206    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-etc-openvswitch\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.253184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252274    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c9e611ab-51d8-4f28-9106-829c3202216c-host-cni-netd\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.253184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252360    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c9e611ab-51d8-4f28-9106-829c3202216c-ovnkube-config\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.253184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252448    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4efaeb6b-b220-4e84-ac19-115ce6ef37e8-konnectivity-ca\") pod \"konnectivity-agent-x6tpp\" (UID: \"4efaeb6b-b220-4e84-ac19-115ce6ef37e8\") " pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:30.253184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.252490    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c9e611ab-51d8-4f28-9106-829c3202216c-ovnkube-script-lib\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.254045 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.254028    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c9e611ab-51d8-4f28-9106-829c3202216c-ovn-node-metrics-cert\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.254818 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.254802    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4efaeb6b-b220-4e84-ac19-115ce6ef37e8-agent-certs\") pod \"konnectivity-agent-x6tpp\" (UID: \"4efaeb6b-b220-4e84-ac19-115ce6ef37e8\") " pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:30.262864 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.262803    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c47q5\" (UniqueName: \"kubernetes.io/projected/c9e611ab-51d8-4f28-9106-829c3202216c-kube-api-access-c47q5\") pod \"ovnkube-node-vmx6b\" (UID: \"c9e611ab-51d8-4f28-9106-829c3202216c\") " pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.337014 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.336982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq"
Jul 09 09:33:30.337337 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.337315    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:30.346985 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.346959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2r6dl"
Jul 09 09:33:30.355746 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.355726    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fxrwr"
Jul 09 09:33:30.362360 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.362345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kwccx"
Jul 09 09:33:30.370037 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.370014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2fd5t"
Jul 09 09:33:30.376647 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.376627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pk5f2"
Jul 09 09:33:30.384467 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.384452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2tnhw"
Jul 09 09:33:30.391087 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.391066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:30.396555 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.396537    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:30.428853 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.428829    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:30.654163 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.654089    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:30.654325 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.654252    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:30.654367 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.654331    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs podName:cc58490a-3112-469d-b967-06c5bd8c9e20 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:31.65431376 +0000 UTC m=+3.979276021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs") pod "network-metrics-daemon-5g56w" (UID: "cc58490a-3112-469d-b967-06c5bd8c9e20") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:30.754642 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.754613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8bt\" (UniqueName: \"kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt\") pod \"network-check-target-9gj6d\" (UID: \"1e83b238-3f02-48c8-a05a-045a2e0fc781\") " pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:30.754780 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:30.754673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:30.754780 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.754766    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:30.754780 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.754770    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:30.754908 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.754790    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:30.754908 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.754800    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sj8bt for pod openshift-network-diagnostics/network-check-target-9gj6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:30.754908 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.754816    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret podName:8aa5ea73-c624-40b0-8672-e1b09f0ee72b nodeName:}" failed. No retries permitted until 2026-07-09 09:33:31.754804384 +0000 UTC m=+4.079766645 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret") pod "global-pull-secret-syncer-wg7tt" (UID: "8aa5ea73-c624-40b0-8672-e1b09f0ee72b") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:30.754908 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:30.754842    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt podName:1e83b238-3f02-48c8-a05a-045a2e0fc781 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:31.754830076 +0000 UTC m=+4.079792338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj8bt" (UniqueName: "kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt") pod "network-check-target-9gj6d" (UID: "1e83b238-3f02-48c8-a05a-045a2e0fc781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:31.078868 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.078764    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 09:28:30 +0000 UTC" deadline="2028-04-25 20:54:19.96160515 +0000 UTC"
Jul 09 09:33:31.078868 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.078810    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15755h20m48.882797855s"
Jul 09 09:33:31.121197 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:31.121173    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod442a1957_0ac5_4abb_a1ea_cc11181c6e5a.slice/crio-b1ae19d1d8b07199d5dca17930581ab6c383c2e888b39fd73f8df9a825246991 WatchSource:0}: Error finding container b1ae19d1d8b07199d5dca17930581ab6c383c2e888b39fd73f8df9a825246991: Status 404 returned error can't find the container with id b1ae19d1d8b07199d5dca17930581ab6c383c2e888b39fd73f8df9a825246991
Jul 09 09:33:31.123138 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:31.123112    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efaeb6b_b220_4e84_ac19_115ce6ef37e8.slice/crio-0bb15340623e5eeb74d12f7d0cbc18c704579e1816efa48a05797af989dce499 WatchSource:0}: Error finding container 0bb15340623e5eeb74d12f7d0cbc18c704579e1816efa48a05797af989dce499: Status 404 returned error can't find the container with id 0bb15340623e5eeb74d12f7d0cbc18c704579e1816efa48a05797af989dce499
Jul 09 09:33:31.126093 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:31.125971    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7320fb87_f8d7_4d0c_9e3a_5da86b108b62.slice/crio-d5f4885ffd2e21eb8f02448969bde5a625c8ee7dec37b3cccf4e269594a1b24d WatchSource:0}: Error finding container d5f4885ffd2e21eb8f02448969bde5a625c8ee7dec37b3cccf4e269594a1b24d: Status 404 returned error can't find the container with id d5f4885ffd2e21eb8f02448969bde5a625c8ee7dec37b3cccf4e269594a1b24d
Jul 09 09:33:31.126771 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:31.126733    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7386da_1749_4189_ab62_665534f4344b.slice/crio-b87803b5d2ba172a63f6637b6fed2888a30469dbfc7b0eadf7ff4fbc235b416a WatchSource:0}: Error finding container b87803b5d2ba172a63f6637b6fed2888a30469dbfc7b0eadf7ff4fbc235b416a: Status 404 returned error can't find the container with id b87803b5d2ba172a63f6637b6fed2888a30469dbfc7b0eadf7ff4fbc235b416a
Jul 09 09:33:31.127465 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:31.127432    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e611ab_51d8_4f28_9106_829c3202216c.slice/crio-8dde53d489d49be9ede03653dd2d3fb563f1df576fc1fbf86cad7b49004443af WatchSource:0}: Error finding container 8dde53d489d49be9ede03653dd2d3fb563f1df576fc1fbf86cad7b49004443af: Status 404 returned error can't find the container with id 8dde53d489d49be9ede03653dd2d3fb563f1df576fc1fbf86cad7b49004443af
Jul 09 09:33:31.128961 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:31.128903    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a09ec5_5646_45b9_bcce_9c96d2e9537c.slice/crio-4ec0aaf9c1df2c856a633ee78f2d3b428e9bf9d6beeec740e450e0721864ed56 WatchSource:0}: Error finding container 4ec0aaf9c1df2c856a633ee78f2d3b428e9bf9d6beeec740e450e0721864ed56: Status 404 returned error can't find the container with id 4ec0aaf9c1df2c856a633ee78f2d3b428e9bf9d6beeec740e450e0721864ed56
Jul 09 09:33:31.130432 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:31.130355    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb0783e_86d2_42bb_96e8_c69c1fae5629.slice/crio-9de53db51217f96c6e633fc00ccf7025a93974608d0fc36d00165c56632bd94d WatchSource:0}: Error finding container 9de53db51217f96c6e633fc00ccf7025a93974608d0fc36d00165c56632bd94d: Status 404 returned error can't find the container with id 9de53db51217f96c6e633fc00ccf7025a93974608d0fc36d00165c56632bd94d
Jul 09 09:33:31.131344 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:31.131151    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf832bbb8_9acb_4a0b_bee3_77a5d5aad1d4.slice/crio-e42a89f576c8fd367327fc883d277566838c2705d93d4df9a6c1f5dc112ebf4f WatchSource:0}: Error finding container e42a89f576c8fd367327fc883d277566838c2705d93d4df9a6c1f5dc112ebf4f: Status 404 returned error can't find the container with id e42a89f576c8fd367327fc883d277566838c2705d93d4df9a6c1f5dc112ebf4f
Jul 09 09:33:31.133774 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:33:31.133753    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1394ed00_564d_4d9d_87f1_8990a5f44301.slice/crio-5ebbaed922dad1e61317e7246fc80a7059ee9f2fd8ffcf34ca0b8a132d1b3c8e WatchSource:0}: Error finding container 5ebbaed922dad1e61317e7246fc80a7059ee9f2fd8ffcf34ca0b8a132d1b3c8e: Status 404 returned error can't find the container with id 5ebbaed922dad1e61317e7246fc80a7059ee9f2fd8ffcf34ca0b8a132d1b3c8e
Jul 09 09:33:31.159707 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.159688    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 09:33:31.223214 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.223188    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:31.223299 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.223227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:31.223337 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:31.223295    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:31.223438 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:31.223419    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:31.234907 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.234858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal" event={"ID":"78255babc9a1b932c1400b78f8b89518","Type":"ContainerStarted","Data":"1e8eddb4398aec6786e5549593ebdb3612f588e7b9873c7bd3d6ac4e1d7ee045"}
Jul 09 09:33:31.237223 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.237163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq" event={"ID":"2f7386da-1749-4189-ab62-665534f4344b","Type":"ContainerStarted","Data":"b87803b5d2ba172a63f6637b6fed2888a30469dbfc7b0eadf7ff4fbc235b416a"}
Jul 09 09:33:31.239418 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.239394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fxrwr" event={"ID":"21a09ec5-5646-45b9-bcce-9c96d2e9537c","Type":"ContainerStarted","Data":"4ec0aaf9c1df2c856a633ee78f2d3b428e9bf9d6beeec740e450e0721864ed56"}
Jul 09 09:33:31.240429 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.240392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2r6dl" event={"ID":"1394ed00-564d-4d9d-87f1-8990a5f44301","Type":"ContainerStarted","Data":"5ebbaed922dad1e61317e7246fc80a7059ee9f2fd8ffcf34ca0b8a132d1b3c8e"}
Jul 09 09:33:31.241364 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.241344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pk5f2" event={"ID":"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4","Type":"ContainerStarted","Data":"e42a89f576c8fd367327fc883d277566838c2705d93d4df9a6c1f5dc112ebf4f"}
Jul 09 09:33:31.242365 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.242344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2fd5t" event={"ID":"8eb0783e-86d2-42bb-96e8-c69c1fae5629","Type":"ContainerStarted","Data":"9de53db51217f96c6e633fc00ccf7025a93974608d0fc36d00165c56632bd94d"}
Jul 09 09:33:31.243585 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.243552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerStarted","Data":"8dde53d489d49be9ede03653dd2d3fb563f1df576fc1fbf86cad7b49004443af"}
Jul 09 09:33:31.244674 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.244649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tnhw" event={"ID":"7320fb87-f8d7-4d0c-9e3a-5da86b108b62","Type":"ContainerStarted","Data":"d5f4885ffd2e21eb8f02448969bde5a625c8ee7dec37b3cccf4e269594a1b24d"}
Jul 09 09:33:31.245792 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.245769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x6tpp" event={"ID":"4efaeb6b-b220-4e84-ac19-115ce6ef37e8","Type":"ContainerStarted","Data":"0bb15340623e5eeb74d12f7d0cbc18c704579e1816efa48a05797af989dce499"}
Jul 09 09:33:31.246782 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.246760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kwccx" event={"ID":"442a1957-0ac5-4abb-a1ea-cc11181c6e5a","Type":"ContainerStarted","Data":"b1ae19d1d8b07199d5dca17930581ab6c383c2e888b39fd73f8df9a825246991"}
Jul 09 09:33:31.247005 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.246962    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-157.ec2.internal" podStartSLOduration=2.246944859 podStartE2EDuration="2.246944859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:31.246658523 +0000 UTC m=+3.571620808" watchObservedRunningTime="2026-07-09 09:33:31.246944859 +0000 UTC m=+3.571907144"
Jul 09 09:33:31.662490 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.662453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:31.662642 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:31.662592    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:31.662696 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:31.662654    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs podName:cc58490a-3112-469d-b967-06c5bd8c9e20 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:33.66263518 +0000 UTC m=+5.987597442 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs") pod "network-metrics-daemon-5g56w" (UID: "cc58490a-3112-469d-b967-06c5bd8c9e20") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:31.764144 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.763299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:31.764144 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:31.763349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8bt\" (UniqueName: \"kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt\") pod \"network-check-target-9gj6d\" (UID: \"1e83b238-3f02-48c8-a05a-045a2e0fc781\") " pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:31.764144 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:31.763547    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:31.764144 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:31.763567    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:31.764144 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:31.763579    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sj8bt for pod openshift-network-diagnostics/network-check-target-9gj6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:31.764144 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:31.763636    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt podName:1e83b238-3f02-48c8-a05a-045a2e0fc781 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:33.763617912 +0000 UTC m=+6.088580192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj8bt" (UniqueName: "kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt") pod "network-check-target-9gj6d" (UID: "1e83b238-3f02-48c8-a05a-045a2e0fc781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:31.764144 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:31.764059    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:31.764144 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:31.764109    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret podName:8aa5ea73-c624-40b0-8672-e1b09f0ee72b nodeName:}" failed. No retries permitted until 2026-07-09 09:33:33.764094457 +0000 UTC m=+6.089056721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret") pod "global-pull-secret-syncer-wg7tt" (UID: "8aa5ea73-c624-40b0-8672-e1b09f0ee72b") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:32.079814 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:32.079700    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 09:28:30 +0000 UTC" deadline="2027-12-22 17:59:10.036667386 +0000 UTC"
Jul 09 09:33:32.079814 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:32.079760    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12752h25m37.956910667s"
Jul 09 09:33:32.225788 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:32.225756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:32.225951 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:32.225906    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:32.282155 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:32.282121    2581 generic.go:356] "Generic (PLEG): container finished" podID="d2c0eff54d21d6540cf23785ec244815" containerID="13de8d6620257b6e3318ec2c3e6532f38655df3d2df965658637713c5e965dae" exitCode=0
Jul 09 09:33:32.282340 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:32.282319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal" event={"ID":"d2c0eff54d21d6540cf23785ec244815","Type":"ContainerDied","Data":"13de8d6620257b6e3318ec2c3e6532f38655df3d2df965658637713c5e965dae"}
Jul 09 09:33:33.222788 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:33.222757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:33.223237 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:33.222899    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:33.223319 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:33.223253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:33.223403 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:33.223360    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:33.306504 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:33.306469    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal" event={"ID":"d2c0eff54d21d6540cf23785ec244815","Type":"ContainerStarted","Data":"85424601378badec36667d0cfe9d989a02d49719eb7c39ffc38c12842dc454fa"}
Jul 09 09:33:33.320964 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:33.320908    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-157.ec2.internal" podStartSLOduration=4.320868101 podStartE2EDuration="4.320868101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:33.320538556 +0000 UTC m=+5.645500842" watchObservedRunningTime="2026-07-09 09:33:33.320868101 +0000 UTC m=+5.645830377"
Jul 09 09:33:33.681082 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:33.680989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:33.681242 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:33.681141    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:33.681242 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:33.681210    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs podName:cc58490a-3112-469d-b967-06c5bd8c9e20 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:37.681191846 +0000 UTC m=+10.006154112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs") pod "network-metrics-daemon-5g56w" (UID: "cc58490a-3112-469d-b967-06c5bd8c9e20") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:33.782283 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:33.782243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:33.782474 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:33.782308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8bt\" (UniqueName: \"kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt\") pod \"network-check-target-9gj6d\" (UID: \"1e83b238-3f02-48c8-a05a-045a2e0fc781\") " pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:33.782548 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:33.782480    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:33.782548 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:33.782498    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:33.782548 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:33.782511    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sj8bt for pod openshift-network-diagnostics/network-check-target-9gj6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:33.782702 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:33.782570    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt podName:1e83b238-3f02-48c8-a05a-045a2e0fc781 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:37.78255144 +0000 UTC m=+10.107513722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj8bt" (UniqueName: "kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt") pod "network-check-target-9gj6d" (UID: "1e83b238-3f02-48c8-a05a-045a2e0fc781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:33.782702 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:33.782640    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:33.782702 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:33.782675    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret podName:8aa5ea73-c624-40b0-8672-e1b09f0ee72b nodeName:}" failed. No retries permitted until 2026-07-09 09:33:37.782664064 +0000 UTC m=+10.107626332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret") pod "global-pull-secret-syncer-wg7tt" (UID: "8aa5ea73-c624-40b0-8672-e1b09f0ee72b") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:34.223076 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:34.223034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:34.223513 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:34.223178    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:35.223512 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:35.223475    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:35.223991 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:35.223600    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:35.223991 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:35.223691    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:35.223991 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:35.223772    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:36.223388 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:36.223290    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:36.223557 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:36.223411    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:37.223450 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:37.223416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:37.223628 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:37.223536    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:37.224042 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:37.223416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:37.224042 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:37.224000    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:37.717046 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:37.716951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:37.717229 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:37.717098    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:37.717229 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:37.717162    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs podName:cc58490a-3112-469d-b967-06c5bd8c9e20 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:45.717143461 +0000 UTC m=+18.042105737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs") pod "network-metrics-daemon-5g56w" (UID: "cc58490a-3112-469d-b967-06c5bd8c9e20") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:37.817478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:37.817434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:37.817673 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:37.817492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8bt\" (UniqueName: \"kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt\") pod \"network-check-target-9gj6d\" (UID: \"1e83b238-3f02-48c8-a05a-045a2e0fc781\") " pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:37.817735 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:37.817670    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:37.817735 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:37.817688    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:37.817735 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:37.817700    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sj8bt for pod openshift-network-diagnostics/network-check-target-9gj6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:37.817924 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:37.817761    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt podName:1e83b238-3f02-48c8-a05a-045a2e0fc781 nodeName:}" failed. No retries permitted until 2026-07-09 09:33:45.817741542 +0000 UTC m=+18.142703806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj8bt" (UniqueName: "kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt") pod "network-check-target-9gj6d" (UID: "1e83b238-3f02-48c8-a05a-045a2e0fc781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:37.818276 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:37.818246    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:37.818371 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:37.818319    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret podName:8aa5ea73-c624-40b0-8672-e1b09f0ee72b nodeName:}" failed. No retries permitted until 2026-07-09 09:33:45.818301868 +0000 UTC m=+18.143264135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret") pod "global-pull-secret-syncer-wg7tt" (UID: "8aa5ea73-c624-40b0-8672-e1b09f0ee72b") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:38.223465 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:38.223434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:38.224432 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:38.224396    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:39.223175 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:39.223142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:39.223363 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:39.223268    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:39.223363 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:39.223277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:39.223484 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:39.223369    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:40.223492 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:40.223368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:40.223995 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:40.223505    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:41.222953 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:41.222905    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:41.223137 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:41.222963    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:41.223137 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:41.223032    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:41.223251 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:41.223145    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:42.223017 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:42.222984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:42.223405 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:42.223109    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:43.222868 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:43.222836    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:43.223049 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:43.222836    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:43.223049 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:43.222989    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:43.223049 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:43.223039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:44.223080 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:44.223046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:44.223545 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:44.223188    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:45.223354 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:45.223298    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:45.223354 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:45.223338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:45.223920 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:45.223453    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:45.223920 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:45.223578    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:45.777908 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:45.777868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:45.778140 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:45.778010    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:45.778140 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:45.778079    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs podName:cc58490a-3112-469d-b967-06c5bd8c9e20 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:01.778061631 +0000 UTC m=+34.103023893 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs") pod "network-metrics-daemon-5g56w" (UID: "cc58490a-3112-469d-b967-06c5bd8c9e20") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:33:45.879196 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:45.879166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8bt\" (UniqueName: \"kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt\") pod \"network-check-target-9gj6d\" (UID: \"1e83b238-3f02-48c8-a05a-045a2e0fc781\") " pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:45.879354 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:45.879238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:45.879354 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:45.879316    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:45.879354 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:45.879320    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:33:45.879354 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:45.879339    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:33:45.879354 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:45.879349    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sj8bt for pod openshift-network-diagnostics/network-check-target-9gj6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:45.879504 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:45.879372    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret podName:8aa5ea73-c624-40b0-8672-e1b09f0ee72b nodeName:}" failed. No retries permitted until 2026-07-09 09:34:01.879358769 +0000 UTC m=+34.204321031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret") pod "global-pull-secret-syncer-wg7tt" (UID: "8aa5ea73-c624-40b0-8672-e1b09f0ee72b") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:33:45.879504 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:45.879390    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt podName:1e83b238-3f02-48c8-a05a-045a2e0fc781 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:01.879378011 +0000 UTC m=+34.204340277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj8bt" (UniqueName: "kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt") pod "network-check-target-9gj6d" (UID: "1e83b238-3f02-48c8-a05a-045a2e0fc781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:33:46.223430 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:46.223397    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:46.223825 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:46.223536    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:47.223636 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:47.223604    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:47.223636 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:47.223629    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:47.224127 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:47.223721    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:47.224127 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:47.223851    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:48.224076 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:48.224046    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:48.224601 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:48.224136    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:49.223374 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:49.223349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:49.223541 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:49.223348    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:49.223541 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:49.223458    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:49.223541 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:49.223517    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:50.222644 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.222594    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:50.222964 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:50.222723    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:50.342260 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.342091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerStarted","Data":"4cf0a0d4f9bd3de0d5f4e0c30c4c47c26e392b7695b6610968e059d5c5c48a69"}
Jul 09 09:33:50.342367 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.342323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerStarted","Data":"d8de8ec027521b837827efdffaa72d4ef7f15eb38a35d22305a21b566189301a"}
Jul 09 09:33:50.343359 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.343334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tnhw" event={"ID":"7320fb87-f8d7-4d0c-9e3a-5da86b108b62","Type":"ContainerStarted","Data":"876a59ae351b382bd48d70f2f34a9e7f3f0f23a1baaa0cc621655bdb2cbf4958"}
Jul 09 09:33:50.344672 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.344653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x6tpp" event={"ID":"4efaeb6b-b220-4e84-ac19-115ce6ef37e8","Type":"ContainerStarted","Data":"1a66f561a54bf4398fe9adaa0d90ed41405b48c98b33abdabea5ea311d0f4893"}
Jul 09 09:33:50.345988 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.345961    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kwccx" event={"ID":"442a1957-0ac5-4abb-a1ea-cc11181c6e5a","Type":"ContainerStarted","Data":"765c92b025b4f47025147eb798705b76395baad7849b72bc358ff00ea34e51dc"}
Jul 09 09:33:50.347132 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.347114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq" event={"ID":"2f7386da-1749-4189-ab62-665534f4344b","Type":"ContainerStarted","Data":"ae1c810864252632aa79ff2f2f1e679db7614b09dba56d431371e7c27ed5f3b5"}
Jul 09 09:33:50.348349 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.348329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pk5f2" event={"ID":"f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4","Type":"ContainerStarted","Data":"319a0cd2e66d14dbde7cac7481b12023d8e6e06640adac11f5f77ca477f5bf91"}
Jul 09 09:33:50.349526 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.349507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2fd5t" event={"ID":"8eb0783e-86d2-42bb-96e8-c69c1fae5629","Type":"ContainerStarted","Data":"d3921a27a3d363c1a0703c037c4e0b1ba9538a6f0147eb5e2e6a26a5b4996f1d"}
Jul 09 09:33:50.395403 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.395363    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kwccx" podStartSLOduration=3.48761793 podStartE2EDuration="22.395347975s" totalImagesPullingTime="18.907730045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:50.380810828 +0000 UTC m=+22.705773111" watchObservedRunningTime="2026-07-09 09:33:50.395347975 +0000 UTC m=+22.720310259"
Jul 09 09:33:50.395501 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.395454    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2fd5t" podStartSLOduration=3.674417149 podStartE2EDuration="22.395450294s" totalImagesPullingTime="18.721033145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:50.394396334 +0000 UTC m=+22.719358619" watchObservedRunningTime="2026-07-09 09:33:50.395450294 +0000 UTC m=+22.720412578"
Jul 09 09:33:50.411985 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.410348    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-pk5f2" podStartSLOduration=3.689106618 podStartE2EDuration="22.410327021s" totalImagesPullingTime="18.721220403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:50.409909946 +0000 UTC m=+22.734872244" watchObservedRunningTime="2026-07-09 09:33:50.410327021 +0000 UTC m=+22.735289306"
Jul 09 09:33:50.428453 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:50.428394    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x6tpp" podStartSLOduration=7.7611288720000005 podStartE2EDuration="22.428370739s" totalImagesPullingTime="14.667241867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:50.428001115 +0000 UTC m=+22.752963400" watchObservedRunningTime="2026-07-09 09:33:50.428370739 +0000 UTC m=+22.753333024"
Jul 09 09:33:51.223067 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.222866    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:51.223804 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.222876    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:51.223804 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:51.223180    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:51.223804 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:51.223225    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:51.353589 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.353557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fxrwr" event={"ID":"21a09ec5-5646-45b9-bcce-9c96d2e9537c","Type":"ContainerStarted","Data":"82616aa13d9909bb39d04eacc2a83a84257cc314af23cd44fd368e6f5f70ca29"}
Jul 09 09:33:51.355241 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.355208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2r6dl" event={"ID":"1394ed00-564d-4d9d-87f1-8990a5f44301","Type":"ContainerStarted","Data":"e60257fb2ce3b486382e6c524a5148d20b51c944cf78cced176acb7333eb4699"}
Jul 09 09:33:51.358230 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.358208    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:33:51.358608 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.358574    2581 generic.go:356] "Generic (PLEG): container finished" podID="c9e611ab-51d8-4f28-9106-829c3202216c" containerID="4cf0a0d4f9bd3de0d5f4e0c30c4c47c26e392b7695b6610968e059d5c5c48a69" exitCode=1
Jul 09 09:33:51.358670 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.358606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerDied","Data":"4cf0a0d4f9bd3de0d5f4e0c30c4c47c26e392b7695b6610968e059d5c5c48a69"}
Jul 09 09:33:51.358720 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.358687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerStarted","Data":"e558391a4581a1b96f41d35a600fd64d37a6b2a09ad1d6bf54b63825e2a06bd9"}
Jul 09 09:33:51.358778 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.358735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerStarted","Data":"1568288700d5b338bd414e509ad611e4b8107886b6907fbab4dfaa41ac836dbe"}
Jul 09 09:33:51.358829 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.358782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerStarted","Data":"5ecf146171fa394dcc2ac255407fb83b7b9e65407c9db0a0f8558b57aac3c31a"}
Jul 09 09:33:51.358878 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.358827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerStarted","Data":"611ac2b56bdedd52ff94ba82a91e17098ff781434c33a47ca1ac9f5a885aaa96"}
Jul 09 09:33:51.360171 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.360149    2581 generic.go:356] "Generic (PLEG): container finished" podID="7320fb87-f8d7-4d0c-9e3a-5da86b108b62" containerID="876a59ae351b382bd48d70f2f34a9e7f3f0f23a1baaa0cc621655bdb2cbf4958" exitCode=0
Jul 09 09:33:51.360283 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.360263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tnhw" event={"ID":"7320fb87-f8d7-4d0c-9e3a-5da86b108b62","Type":"ContainerDied","Data":"876a59ae351b382bd48d70f2f34a9e7f3f0f23a1baaa0cc621655bdb2cbf4958"}
Jul 09 09:33:51.374754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.374634    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fxrwr" podStartSLOduration=4.097080313 podStartE2EDuration="23.374618135s" totalImagesPullingTime="19.277537822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:51.37450461 +0000 UTC m=+23.699466894" watchObservedRunningTime="2026-07-09 09:33:51.374618135 +0000 UTC m=+23.699580422"
Jul 09 09:33:51.418923 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.418810    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2r6dl" podStartSLOduration=4.524609316 podStartE2EDuration="23.418790666s" totalImagesPullingTime="18.89418135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:51.418093261 +0000 UTC m=+23.743055545" watchObservedRunningTime="2026-07-09 09:33:51.418790666 +0000 UTC m=+23.743752951"
Jul 09 09:33:51.450155 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:51.450129    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 09:33:52.145361 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:52.145060    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T09:33:51.450145521Z","UUID":"acce0e25-2067-4025-9601-f6ea87f1184e","Handler":null,"Name":"","Endpoint":""}
Jul 09 09:33:52.146909 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:52.146866    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 09:33:52.147037 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:52.146956    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 09:33:52.223229 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:52.223198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:52.223786 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:52.223337    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:52.363745 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:52.363707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq" event={"ID":"2f7386da-1749-4189-ab62-665534f4344b","Type":"ContainerStarted","Data":"e346678f75e5e84156ecf4cd5f940fcb992f3d33ca98f511187af9758951479c"}
Jul 09 09:33:53.223548 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:53.223520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:53.224064 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:53.223522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:53.224064 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:53.223641    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:53.224064 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:53.223720    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:53.369595 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:53.369565    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:33:53.370138 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:53.370103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerStarted","Data":"f6c08e76426bca3abdaf8ba6a13fe857e4188a66f84d71d9666660404619d113"}
Jul 09 09:33:53.372423 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:53.372396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq" event={"ID":"2f7386da-1749-4189-ab62-665534f4344b","Type":"ContainerStarted","Data":"63f5ab9fac5e10a5533eadbf21d935c23790b61944b19981a3b7df9f7ea68d0b"}
Jul 09 09:33:53.393582 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:53.393534    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kn9kq" podStartSLOduration=4.578469005 podStartE2EDuration="25.393520363s" totalImagesPullingTime="20.815051358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:53.392877083 +0000 UTC m=+25.717839367" watchObservedRunningTime="2026-07-09 09:33:53.393520363 +0000 UTC m=+25.718482647"
Jul 09 09:33:54.226441 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:54.226289    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:54.226781 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:54.226512    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:55.091074 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:55.091041    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:55.091662 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:55.091640    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:55.223368 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:55.223336    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:55.223368 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:55.223352    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:55.223597 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:55.223460    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:55.223653 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:55.223615    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:55.376588 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:55.376555    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:55.377089 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:55.377066    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x6tpp"
Jul 09 09:33:56.223627 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:56.223467    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:56.223801 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:56.223695    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:56.381139 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:56.381047    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:33:56.382003 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:56.381441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerStarted","Data":"c9cccd5be9171feca77f50f4fbf5366ddbfd7149c673c9564f0f09023bf5766e"}
Jul 09 09:33:56.382003 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:56.381681    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:56.382003 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:56.381736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:56.382003 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:56.381850    2581 scope.go:117] "RemoveContainer" containerID="4cf0a0d4f9bd3de0d5f4e0c30c4c47c26e392b7695b6610968e059d5c5c48a69"
Jul 09 09:33:56.383332 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:56.383312    2581 generic.go:356] "Generic (PLEG): container finished" podID="7320fb87-f8d7-4d0c-9e3a-5da86b108b62" containerID="9b63f13b2d8facbfb5f11079c733491aec80c72dd09e9e95fd99f9b22f986e38" exitCode=0
Jul 09 09:33:56.383417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:56.383387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tnhw" event={"ID":"7320fb87-f8d7-4d0c-9e3a-5da86b108b62","Type":"ContainerDied","Data":"9b63f13b2d8facbfb5f11079c733491aec80c72dd09e9e95fd99f9b22f986e38"}
Jul 09 09:33:56.397478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:56.397461    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:57.223456 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:57.223426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:57.223456 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:57.223457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:57.223708 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:57.223547    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:57.223708 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:57.223670    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:57.389586 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:57.389561    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:33:57.389976 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:57.389938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" event={"ID":"c9e611ab-51d8-4f28-9106-829c3202216c","Type":"ContainerStarted","Data":"2e8c2207f81fb16a9f197a613942cd3e5fbd6ecd6fffca20c65e4a9e151f844f"}
Jul 09 09:33:57.390251 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:57.390224    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:57.405000 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:57.404980    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:33:57.429997 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:57.429950    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b" podStartSLOduration=10.529202139 podStartE2EDuration="29.42993602s" totalImagesPullingTime="18.900733881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:33:57.426148513 +0000 UTC m=+29.751110796" watchObservedRunningTime="2026-07-09 09:33:57.42993602 +0000 UTC m=+29.754898304"
Jul 09 09:33:58.032049 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:58.032012    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wg7tt"]
Jul 09 09:33:58.032203 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:58.032123    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:58.032274 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:58.032230    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:58.035159 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:58.035121    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5g56w"]
Jul 09 09:33:58.035360 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:58.035212    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:33:58.035360 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:58.035309    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:33:58.039596 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:58.039574    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9gj6d"]
Jul 09 09:33:58.039725 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:58.039706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:58.039818 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:58.039796    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:33:58.394678 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:58.394642    2581 generic.go:356] "Generic (PLEG): container finished" podID="7320fb87-f8d7-4d0c-9e3a-5da86b108b62" containerID="82a51b25374933c838ccfb80ff79f50778ed26a50e9cb8cd246ff5ab88a6c13e" exitCode=0
Jul 09 09:33:58.395181 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:58.394727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tnhw" event={"ID":"7320fb87-f8d7-4d0c-9e3a-5da86b108b62","Type":"ContainerDied","Data":"82a51b25374933c838ccfb80ff79f50778ed26a50e9cb8cd246ff5ab88a6c13e"}
Jul 09 09:33:59.223319 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:59.223293    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:33:59.223486 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:33:59.223293    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:33:59.223486 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:59.223393    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:33:59.223486 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:33:59.223463    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:34:00.223151 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:00.222981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:34:00.223575 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:00.223237    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:34:00.401840 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:00.401803    2581 generic.go:356] "Generic (PLEG): container finished" podID="7320fb87-f8d7-4d0c-9e3a-5da86b108b62" containerID="74281d66e268a63f89579d3a0c6e8c065d381d5f9f9f4e3109566377fe9319b0" exitCode=0
Jul 09 09:34:00.402007 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:00.401866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tnhw" event={"ID":"7320fb87-f8d7-4d0c-9e3a-5da86b108b62","Type":"ContainerDied","Data":"74281d66e268a63f89579d3a0c6e8c065d381d5f9f9f4e3109566377fe9319b0"}
Jul 09 09:34:01.223428 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:01.223396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:34:01.223931 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:01.223396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:34:01.223931 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:01.223534    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wg7tt" podUID="8aa5ea73-c624-40b0-8672-e1b09f0ee72b"
Jul 09 09:34:01.223931 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:01.223601    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9gj6d" podUID="1e83b238-3f02-48c8-a05a-045a2e0fc781"
Jul 09 09:34:01.801255 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:01.801219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:34:01.801426 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:01.801334    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:34:01.801426 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:01.801392    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs podName:cc58490a-3112-469d-b967-06c5bd8c9e20 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:33.801375328 +0000 UTC m=+66.126337592 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs") pod "network-metrics-daemon-5g56w" (UID: "cc58490a-3112-469d-b967-06c5bd8c9e20") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 09:34:01.901766 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:01.901716    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:34:01.901973 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:01.901777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8bt\" (UniqueName: \"kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt\") pod \"network-check-target-9gj6d\" (UID: \"1e83b238-3f02-48c8-a05a-045a2e0fc781\") " pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:34:01.901973 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:01.901913    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 09:34:01.901973 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:01.901961    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 09:34:01.902115 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:01.901980    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 09:34:01.902115 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:01.901992    2581 projected.go:196] Error preparing data for projected volume kube-api-access-sj8bt for pod openshift-network-diagnostics/network-check-target-9gj6d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:34:01.902115 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:01.901996    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret podName:8aa5ea73-c624-40b0-8672-e1b09f0ee72b nodeName:}" failed. No retries permitted until 2026-07-09 09:34:33.901972728 +0000 UTC m=+66.226934996 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret") pod "global-pull-secret-syncer-wg7tt" (UID: "8aa5ea73-c624-40b0-8672-e1b09f0ee72b") : object "kube-system"/"original-pull-secret" not registered
Jul 09 09:34:01.902115 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:01.902044    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt podName:1e83b238-3f02-48c8-a05a-045a2e0fc781 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:33.902023868 +0000 UTC m=+66.226986147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj8bt" (UniqueName: "kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt") pod "network-check-target-9gj6d" (UID: "1e83b238-3f02-48c8-a05a-045a2e0fc781") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 09:34:02.223494 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:02.223466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:34:02.223975 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:02.223716    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5g56w" podUID="cc58490a-3112-469d-b967-06c5bd8c9e20"
Jul 09 09:34:02.929673 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:02.929640    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-157.ec2.internal" event="NodeReady"
Jul 09 09:34:02.929848 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:02.929834    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 09:34:03.002374 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.002334    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6ccb779584-t52jv"]
Jul 09 09:34:03.006491 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.006461    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.010322 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.010297    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 09:34:03.010322 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.010297    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pjjd7\""
Jul 09 09:34:03.010814 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.010788    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 09:34:03.011065 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.011047    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 09:34:03.016319 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.016299    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 09:34:03.032810 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.032787    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ccb779584-t52jv"]
Jul 09 09:34:03.037062 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.037042    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-whkx9"]
Jul 09 09:34:03.039185 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.039166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-whkx9"
Jul 09 09:34:03.042909 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.042876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 09:34:03.043143 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.043126    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g7jnx\""
Jul 09 09:34:03.043218 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.043168    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 09:34:03.043644 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.043625    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 09:34:03.050757 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.050736    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fpvfk"]
Jul 09 09:34:03.052604 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.052588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.059950 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.059925    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 09:34:03.060046 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.059975    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 09:34:03.060712 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.060695    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cfzpw\""
Jul 09 09:34:03.066538 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.066520    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-whkx9"]
Jul 09 09:34:03.082429 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.082408    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fpvfk"]
Jul 09 09:34:03.110223 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.110203    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.110343 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.110232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-bound-sa-token\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.110343 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.110271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-ca-trust-extracted\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.110459 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.110351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-trusted-ca\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.110459 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.110397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhkqk\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-kube-api-access-rhkqk\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.110459 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.110439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-installation-pull-secrets\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.110576 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.110473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-image-registry-private-configuration\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.110576 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.110495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-certificates\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.211173 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjnr\" (UniqueName: \"kubernetes.io/projected/bec59435-ae5d-4001-a28c-4b475d5559d0-kube-api-access-nhjnr\") pod \"ingress-canary-whkx9\" (UID: \"bec59435-ae5d-4001-a28c-4b475d5559d0\") " pod="openshift-ingress-canary/ingress-canary-whkx9"
Jul 09 09:34:03.211173 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211175    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.211409 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-bound-sa-token\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.211409 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.211409 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.211305    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 09:34:03.211409 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.211320    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccb779584-t52jv: secret "image-registry-tls" not found
Jul 09 09:34:03.211409 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-tmp-dir\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.211409 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj6gc\" (UniqueName: \"kubernetes.io/projected/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-kube-api-access-mj6gc\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.211409 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.211389    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls podName:f0a3548d-6a6b-45c6-ae75-94c83e693f7c nodeName:}" failed. No retries permitted until 2026-07-09 09:34:03.711367299 +0000 UTC m=+36.036329564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls") pod "image-registry-6ccb779584-t52jv" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c") : secret "image-registry-tls" not found
Jul 09 09:34:03.211763 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-ca-trust-extracted\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.211763 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-trusted-ca\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.211763 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhkqk\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-kube-api-access-rhkqk\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.211763 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-installation-pull-secrets\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.211763 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-image-registry-private-configuration\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.211763 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211585    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-certificates\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.211763 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bec59435-ae5d-4001-a28c-4b475d5559d0-cert\") pod \"ingress-canary-whkx9\" (UID: \"bec59435-ae5d-4001-a28c-4b475d5559d0\") " pod="openshift-ingress-canary/ingress-canary-whkx9"
Jul 09 09:34:03.211763 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-config-volume\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.212166 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.211859    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-ca-trust-extracted\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.212272 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.212252    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-certificates\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.212488 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.212456    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-trusted-ca\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.215904 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.215868    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-installation-pull-secrets\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.216004 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.215986    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-image-registry-private-configuration\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.222807 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.222784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:34:03.222938 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.222917    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:34:03.228021 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.228001    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 09:34:03.228478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.228287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 09:34:03.229259 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.229236    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 09:34:03.229941 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.229921    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhkqk\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-kube-api-access-rhkqk\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.230773 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.230753    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-bound-sa-token\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.231691 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.231666    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r7trk\""
Jul 09 09:34:03.312992 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.312911    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.312992 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.312959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-tmp-dir\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.312992 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.312987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj6gc\" (UniqueName: \"kubernetes.io/projected/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-kube-api-access-mj6gc\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.313252 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.313048    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bec59435-ae5d-4001-a28c-4b475d5559d0-cert\") pod \"ingress-canary-whkx9\" (UID: \"bec59435-ae5d-4001-a28c-4b475d5559d0\") " pod="openshift-ingress-canary/ingress-canary-whkx9"
Jul 09 09:34:03.313252 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.313078    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 09:34:03.313252 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.313101    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-config-volume\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.313252 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.313140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjnr\" (UniqueName: \"kubernetes.io/projected/bec59435-ae5d-4001-a28c-4b475d5559d0-kube-api-access-nhjnr\") pod \"ingress-canary-whkx9\" (UID: \"bec59435-ae5d-4001-a28c-4b475d5559d0\") " pod="openshift-ingress-canary/ingress-canary-whkx9"
Jul 09 09:34:03.313252 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.313168    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls podName:9ffe7afe-800e-4a88-a39d-2b9bd0433a87 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:03.813139254 +0000 UTC m=+36.138101515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls") pod "dns-default-fpvfk" (UID: "9ffe7afe-800e-4a88-a39d-2b9bd0433a87") : secret "dns-default-metrics-tls" not found
Jul 09 09:34:03.313516 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.313349    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-tmp-dir\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.313690 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.313664    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-config-volume\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.315826 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.315804    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bec59435-ae5d-4001-a28c-4b475d5559d0-cert\") pod \"ingress-canary-whkx9\" (UID: \"bec59435-ae5d-4001-a28c-4b475d5559d0\") " pod="openshift-ingress-canary/ingress-canary-whkx9"
Jul 09 09:34:03.324749 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.324729    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjnr\" (UniqueName: \"kubernetes.io/projected/bec59435-ae5d-4001-a28c-4b475d5559d0-kube-api-access-nhjnr\") pod \"ingress-canary-whkx9\" (UID: \"bec59435-ae5d-4001-a28c-4b475d5559d0\") " pod="openshift-ingress-canary/ingress-canary-whkx9"
Jul 09 09:34:03.325061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.325040    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj6gc\" (UniqueName: \"kubernetes.io/projected/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-kube-api-access-mj6gc\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.349007 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.348975    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-whkx9"
Jul 09 09:34:03.715997 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.715958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:03.716185 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.716142    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 09:34:03.716185 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.716166    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccb779584-t52jv: secret "image-registry-tls" not found
Jul 09 09:34:03.716276 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.716235    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls podName:f0a3548d-6a6b-45c6-ae75-94c83e693f7c nodeName:}" failed. No retries permitted until 2026-07-09 09:34:04.716214681 +0000 UTC m=+37.041176949 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls") pod "image-registry-6ccb779584-t52jv" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c") : secret "image-registry-tls" not found
Jul 09 09:34:03.723144 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.723121    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sn8f2"]
Jul 09 09:34:03.725298 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.725280    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.728013 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.727990    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 09:34:03.728612 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.728586    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 09:34:03.728720 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.728627    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 09:34:03.728787 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.728755    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 09:34:03.729085 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.728969    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zwk2d\""
Jul 09 09:34:03.740651 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.740494    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sn8f2"]
Jul 09 09:34:03.817420 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.817384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h2gj\" (UniqueName: \"kubernetes.io/projected/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-api-access-8h2gj\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.817590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.817432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c912314-2d73-4cdb-91d1-f1f7eb547f53-crio-socket\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.817590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.817518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c912314-2d73-4cdb-91d1-f1f7eb547f53-data-volume\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.817590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.817564    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.817748 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.817637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.817748 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.817707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:03.817851 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.817834    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 09:34:03.817950 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.817929    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls podName:9ffe7afe-800e-4a88-a39d-2b9bd0433a87 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:04.817908477 +0000 UTC m=+37.142870739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls") pod "dns-default-fpvfk" (UID: "9ffe7afe-800e-4a88-a39d-2b9bd0433a87") : secret "dns-default-metrics-tls" not found
Jul 09 09:34:03.918305 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.918266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.918479 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.918329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.918479 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.918411    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 09:34:03.918479 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.918431    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 09:34:03.918634 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.918516    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm podName:9c912314-2d73-4cdb-91d1-f1f7eb547f53 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:04.418493379 +0000 UTC m=+36.743455648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sn8f2" (UID: "9c912314-2d73-4cdb-91d1-f1f7eb547f53") : configmap "kube-rbac-proxy" not found
Jul 09 09:34:03.918634 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.918540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h2gj\" (UniqueName: \"kubernetes.io/projected/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-api-access-8h2gj\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.918634 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:03.918577    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls podName:9c912314-2d73-4cdb-91d1-f1f7eb547f53 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:04.418551488 +0000 UTC m=+36.743513768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sn8f2" (UID: "9c912314-2d73-4cdb-91d1-f1f7eb547f53") : secret "insights-runtime-extractor-tls" not found
Jul 09 09:34:03.918634 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.918602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c912314-2d73-4cdb-91d1-f1f7eb547f53-crio-socket\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.918806 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.918641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c912314-2d73-4cdb-91d1-f1f7eb547f53-data-volume\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.918903 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.918867    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9c912314-2d73-4cdb-91d1-f1f7eb547f53-crio-socket\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.919042 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.918959    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9c912314-2d73-4cdb-91d1-f1f7eb547f53-data-volume\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:03.927225 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:03.927206    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h2gj\" (UniqueName: \"kubernetes.io/projected/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-api-access-8h2gj\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:04.196019 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.195946    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c"]
Jul 09 09:34:04.198603 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.198584    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c"
Jul 09 09:34:04.201567 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.201548    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xqc77\""
Jul 09 09:34:04.202133 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.202110    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 09:34:04.203099 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.203081    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 09:34:04.208176 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.208153    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c"]
Jul 09 09:34:04.223475 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.223459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:34:04.225832 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.225813    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 09:34:04.226061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.226045    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ntnhc\""
Jul 09 09:34:04.321383 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.321348    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8rgc\" (UniqueName: \"kubernetes.io/projected/deaaf287-25fd-457b-95ed-1cc34da4a9d1-kube-api-access-l8rgc\") pod \"migrator-d57d58cc9-kw85c\" (UID: \"deaaf287-25fd-457b-95ed-1cc34da4a9d1\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c"
Jul 09 09:34:04.422500 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.422472    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:04.422676 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.422510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8rgc\" (UniqueName: \"kubernetes.io/projected/deaaf287-25fd-457b-95ed-1cc34da4a9d1-kube-api-access-l8rgc\") pod \"migrator-d57d58cc9-kw85c\" (UID: \"deaaf287-25fd-457b-95ed-1cc34da4a9d1\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c"
Jul 09 09:34:04.422676 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.422538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:04.422676 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:04.422615    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 09:34:04.422676 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:04.422667    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 09:34:04.422881 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:04.422693    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm podName:9c912314-2d73-4cdb-91d1-f1f7eb547f53 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:05.422671327 +0000 UTC m=+37.747633590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sn8f2" (UID: "9c912314-2d73-4cdb-91d1-f1f7eb547f53") : configmap "kube-rbac-proxy" not found
Jul 09 09:34:04.422881 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:04.422714    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls podName:9c912314-2d73-4cdb-91d1-f1f7eb547f53 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:05.422704523 +0000 UTC m=+37.747666815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sn8f2" (UID: "9c912314-2d73-4cdb-91d1-f1f7eb547f53") : secret "insights-runtime-extractor-tls" not found
Jul 09 09:34:04.432215 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.432193    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8rgc\" (UniqueName: \"kubernetes.io/projected/deaaf287-25fd-457b-95ed-1cc34da4a9d1-kube-api-access-l8rgc\") pod \"migrator-d57d58cc9-kw85c\" (UID: \"deaaf287-25fd-457b-95ed-1cc34da4a9d1\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c"
Jul 09 09:34:04.508381 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.508350    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c"
Jul 09 09:34:04.629361 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.629178    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-whkx9"]
Jul 09 09:34:04.632250 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:04.632228    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbec59435_ae5d_4001_a28c_4b475d5559d0.slice/crio-87f36e5cf37579f95adb721bdc04f4cf8c3dc34540ad568b4edff3487c286143 WatchSource:0}: Error finding container 87f36e5cf37579f95adb721bdc04f4cf8c3dc34540ad568b4edff3487c286143: Status 404 returned error can't find the container with id 87f36e5cf37579f95adb721bdc04f4cf8c3dc34540ad568b4edff3487c286143
Jul 09 09:34:04.647184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.647160    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c"]
Jul 09 09:34:04.649388 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:04.649357    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeaaf287_25fd_457b_95ed_1cc34da4a9d1.slice/crio-0922dbe5bd46fb9493917a708bf42f47f5736f7a48433e5d09ea9577d714ea9e WatchSource:0}: Error finding container 0922dbe5bd46fb9493917a708bf42f47f5736f7a48433e5d09ea9577d714ea9e: Status 404 returned error can't find the container with id 0922dbe5bd46fb9493917a708bf42f47f5736f7a48433e5d09ea9577d714ea9e
Jul 09 09:34:04.724406 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.724377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:04.724544 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:04.724527    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 09:34:04.724603 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:04.724546    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccb779584-t52jv: secret "image-registry-tls" not found
Jul 09 09:34:04.724646 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:04.724603    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls podName:f0a3548d-6a6b-45c6-ae75-94c83e693f7c nodeName:}" failed. No retries permitted until 2026-07-09 09:34:06.724586708 +0000 UTC m=+39.049548974 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls") pod "image-registry-6ccb779584-t52jv" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c") : secret "image-registry-tls" not found
Jul 09 09:34:04.824904 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:04.824856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:04.825051 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:04.825000    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 09:34:04.825096 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:04.825055    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls podName:9ffe7afe-800e-4a88-a39d-2b9bd0433a87 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:06.825040178 +0000 UTC m=+39.150002440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls") pod "dns-default-fpvfk" (UID: "9ffe7afe-800e-4a88-a39d-2b9bd0433a87") : secret "dns-default-metrics-tls" not found
Jul 09 09:34:05.316409 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.316379    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pk5f2_f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4/dns-node-resolver/0.log"
Jul 09 09:34:05.418638 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.418605    2581 generic.go:356] "Generic (PLEG): container finished" podID="7320fb87-f8d7-4d0c-9e3a-5da86b108b62" containerID="465f94526df1d0e4170ea30b3497f48fb0917bda7fa5a5b9146fde97d35fcea4" exitCode=0
Jul 09 09:34:05.419093 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.418715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tnhw" event={"ID":"7320fb87-f8d7-4d0c-9e3a-5da86b108b62","Type":"ContainerDied","Data":"465f94526df1d0e4170ea30b3497f48fb0917bda7fa5a5b9146fde97d35fcea4"}
Jul 09 09:34:05.420904 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.420863    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c" event={"ID":"deaaf287-25fd-457b-95ed-1cc34da4a9d1","Type":"ContainerStarted","Data":"0922dbe5bd46fb9493917a708bf42f47f5736f7a48433e5d09ea9577d714ea9e"}
Jul 09 09:34:05.423422 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.423376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-whkx9" event={"ID":"bec59435-ae5d-4001-a28c-4b475d5559d0","Type":"ContainerStarted","Data":"87f36e5cf37579f95adb721bdc04f4cf8c3dc34540ad568b4edff3487c286143"}
Jul 09 09:34:05.430337 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.430297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:05.430456 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.430421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:05.430568 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:05.430506    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 09:34:05.430568 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:05.430517    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 09:34:05.430568 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:05.430558    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls podName:9c912314-2d73-4cdb-91d1-f1f7eb547f53 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:07.430541103 +0000 UTC m=+39.755503379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sn8f2" (UID: "9c912314-2d73-4cdb-91d1-f1f7eb547f53") : secret "insights-runtime-extractor-tls" not found
Jul 09 09:34:05.430683 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:05.430573    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm podName:9c912314-2d73-4cdb-91d1-f1f7eb547f53 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:07.430565148 +0000 UTC m=+39.755527410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sn8f2" (UID: "9c912314-2d73-4cdb-91d1-f1f7eb547f53") : configmap "kube-rbac-proxy" not found
Jul 09 09:34:05.969390 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.969363    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-q67lk"]
Jul 09 09:34:05.971875 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.971854    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:05.974780 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.974641    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 09 09:34:05.974780 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.974734    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 09 09:34:05.974780 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.974776    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nl6c7\""
Jul 09 09:34:05.975072 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.975053    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 09 09:34:05.975414 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.975258    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 09 09:34:05.983930 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:05.983862    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-q67lk"]
Jul 09 09:34:06.035284 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.035240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2zrb\" (UniqueName: \"kubernetes.io/projected/e5e28bde-354a-4cb6-a9f1-e8c69987a4f4-kube-api-access-g2zrb\") pod \"service-ca-758f599c5-q67lk\" (UID: \"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4\") " pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:06.035459 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.035441    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e5e28bde-354a-4cb6-a9f1-e8c69987a4f4-signing-cabundle\") pod \"service-ca-758f599c5-q67lk\" (UID: \"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4\") " pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:06.035502 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.035476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e5e28bde-354a-4cb6-a9f1-e8c69987a4f4-signing-key\") pod \"service-ca-758f599c5-q67lk\" (UID: \"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4\") " pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:06.115385 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.115294    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2fd5t_8eb0783e-86d2-42bb-96e8-c69c1fae5629/node-ca/0.log"
Jul 09 09:34:06.136810 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.136624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e5e28bde-354a-4cb6-a9f1-e8c69987a4f4-signing-cabundle\") pod \"service-ca-758f599c5-q67lk\" (UID: \"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4\") " pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:06.136983 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.136818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e5e28bde-354a-4cb6-a9f1-e8c69987a4f4-signing-key\") pod \"service-ca-758f599c5-q67lk\" (UID: \"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4\") " pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:06.136983 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.136841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2zrb\" (UniqueName: \"kubernetes.io/projected/e5e28bde-354a-4cb6-a9f1-e8c69987a4f4-kube-api-access-g2zrb\") pod \"service-ca-758f599c5-q67lk\" (UID: \"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4\") " pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:06.137506 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.137488    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e5e28bde-354a-4cb6-a9f1-e8c69987a4f4-signing-cabundle\") pod \"service-ca-758f599c5-q67lk\" (UID: \"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4\") " pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:06.140904 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.140864    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e5e28bde-354a-4cb6-a9f1-e8c69987a4f4-signing-key\") pod \"service-ca-758f599c5-q67lk\" (UID: \"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4\") " pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:06.145869 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.145845    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2zrb\" (UniqueName: \"kubernetes.io/projected/e5e28bde-354a-4cb6-a9f1-e8c69987a4f4-kube-api-access-g2zrb\") pod \"service-ca-758f599c5-q67lk\" (UID: \"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4\") " pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:06.295382 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.295352    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-q67lk"
Jul 09 09:34:06.429899 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.429814    2581 generic.go:356] "Generic (PLEG): container finished" podID="7320fb87-f8d7-4d0c-9e3a-5da86b108b62" containerID="d66c852adf38f418d34cdbfccc81ded189e076d2f7521027c56e7a802eef5eb8" exitCode=0
Jul 09 09:34:06.430353 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.429933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tnhw" event={"ID":"7320fb87-f8d7-4d0c-9e3a-5da86b108b62","Type":"ContainerDied","Data":"d66c852adf38f418d34cdbfccc81ded189e076d2f7521027c56e7a802eef5eb8"}
Jul 09 09:34:06.432029 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.432005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c" event={"ID":"deaaf287-25fd-457b-95ed-1cc34da4a9d1","Type":"ContainerStarted","Data":"7b13e7495806010bcff6f17739b77877243639876c258e367f3e47fa765510f7"}
Jul 09 09:34:06.432138 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.432050    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c" event={"ID":"deaaf287-25fd-457b-95ed-1cc34da4a9d1","Type":"ContainerStarted","Data":"b36c9a915c32c897b6fd54a6d28d796ab7b3f56987e5fd1367dfe1908e0b7649"}
Jul 09 09:34:06.477767 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.477717    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-kw85c" podStartSLOduration=1.2711149769999999 podStartE2EDuration="2.477699646s" totalImagesPullingTime="1.206584669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:06.476812104 +0000 UTC m=+38.801774392" watchObservedRunningTime="2026-07-09 09:34:06.477699646 +0000 UTC m=+38.802661931"
Jul 09 09:34:06.741757 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.741721    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:06.741980 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:06.741901    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 09:34:06.741980 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:06.741923    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccb779584-t52jv: secret "image-registry-tls" not found
Jul 09 09:34:06.742095 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:06.741988    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls podName:f0a3548d-6a6b-45c6-ae75-94c83e693f7c nodeName:}" failed. No retries permitted until 2026-07-09 09:34:10.741967217 +0000 UTC m=+43.066929483 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls") pod "image-registry-6ccb779584-t52jv" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c") : secret "image-registry-tls" not found
Jul 09 09:34:06.842137 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:06.842108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:06.842282 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:06.842227    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 09:34:06.842328 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:06.842282    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls podName:9ffe7afe-800e-4a88-a39d-2b9bd0433a87 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:10.842266068 +0000 UTC m=+43.167228330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls") pod "dns-default-fpvfk" (UID: "9ffe7afe-800e-4a88-a39d-2b9bd0433a87") : secret "dns-default-metrics-tls" not found
Jul 09 09:34:07.046281 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:07.046221    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-q67lk"]
Jul 09 09:34:07.049523 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:07.049496    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e28bde_354a_4cb6_a9f1_e8c69987a4f4.slice/crio-c26469abfbf24c067f098d0505a60916a8a2ab4222dfc9958251a123757fb2f6 WatchSource:0}: Error finding container c26469abfbf24c067f098d0505a60916a8a2ab4222dfc9958251a123757fb2f6: Status 404 returned error can't find the container with id c26469abfbf24c067f098d0505a60916a8a2ab4222dfc9958251a123757fb2f6
Jul 09 09:34:07.436145 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:07.436110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-whkx9" event={"ID":"bec59435-ae5d-4001-a28c-4b475d5559d0","Type":"ContainerStarted","Data":"605fa2236fea83e7d67669e0b9d94907b38ca979310cb7818607e1fb2637d2e5"}
Jul 09 09:34:07.437345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:07.437323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-q67lk" event={"ID":"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4","Type":"ContainerStarted","Data":"c26469abfbf24c067f098d0505a60916a8a2ab4222dfc9958251a123757fb2f6"}
Jul 09 09:34:07.440872 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:07.440842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2tnhw" event={"ID":"7320fb87-f8d7-4d0c-9e3a-5da86b108b62","Type":"ContainerStarted","Data":"f2ebdb380b6a66e9489d24bba7da058148843440f345a38adeb8ba904b17532e"}
Jul 09 09:34:07.447784 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:07.447759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:07.447881 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:07.447865    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 09:34:07.447951 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:07.447919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:07.447992 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:07.447953    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm podName:9c912314-2d73-4cdb-91d1-f1f7eb547f53 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:11.447927634 +0000 UTC m=+43.772889896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sn8f2" (UID: "9c912314-2d73-4cdb-91d1-f1f7eb547f53") : configmap "kube-rbac-proxy" not found
Jul 09 09:34:07.448031 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:07.448019    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 09:34:07.448087 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:07.448077    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls podName:9c912314-2d73-4cdb-91d1-f1f7eb547f53 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:11.448061229 +0000 UTC m=+43.773023496 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sn8f2" (UID: "9c912314-2d73-4cdb-91d1-f1f7eb547f53") : secret "insights-runtime-extractor-tls" not found
Jul 09 09:34:07.452979 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:07.452856    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-whkx9" podStartSLOduration=1.77912651 podStartE2EDuration="4.452837865s" totalImagesPullingTime="2.673711355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:07.452332939 +0000 UTC m=+39.777295224" watchObservedRunningTime="2026-07-09 09:34:07.452837865 +0000 UTC m=+39.777800151"
Jul 09 09:34:07.475808 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:07.475746    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2tnhw" podStartSLOduration=9.771077707 podStartE2EDuration="39.475728712s" totalImagesPullingTime="29.704651005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:07.47495829 +0000 UTC m=+39.799920575" watchObservedRunningTime="2026-07-09 09:34:07.475728712 +0000 UTC m=+39.800690995"
Jul 09 09:34:10.449635 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:10.449592    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-q67lk" event={"ID":"e5e28bde-354a-4cb6-a9f1-e8c69987a4f4","Type":"ContainerStarted","Data":"45190733f1d68cba70bd19cd4ec3799d5d4448f9cdec9d64acab625797676eb4"}
Jul 09 09:34:10.468454 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:10.468406    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-q67lk" podStartSLOduration=2.867279335 podStartE2EDuration="5.468389316s" totalImagesPullingTime="2.601109981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:10.46679817 +0000 UTC m=+42.791760455" watchObservedRunningTime="2026-07-09 09:34:10.468389316 +0000 UTC m=+42.793351600"
Jul 09 09:34:10.774442 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:10.774402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:10.774592 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:10.774546    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 09:34:10.774592 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:10.774567    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6ccb779584-t52jv: secret "image-registry-tls" not found
Jul 09 09:34:10.774702 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:10.774621    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls podName:f0a3548d-6a6b-45c6-ae75-94c83e693f7c nodeName:}" failed. No retries permitted until 2026-07-09 09:34:18.774606645 +0000 UTC m=+51.099568907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls") pod "image-registry-6ccb779584-t52jv" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c") : secret "image-registry-tls" not found
Jul 09 09:34:10.875726 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:10.875692    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:10.875845 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:10.875828    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 09:34:10.875930 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:10.875920    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls podName:9ffe7afe-800e-4a88-a39d-2b9bd0433a87 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:18.875877129 +0000 UTC m=+51.200839405 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls") pod "dns-default-fpvfk" (UID: "9ffe7afe-800e-4a88-a39d-2b9bd0433a87") : secret "dns-default-metrics-tls" not found
Jul 09 09:34:11.481140 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:11.481105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:11.481620 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:11.481172    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:11.481620 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:11.481299    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 09:34:11.481620 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:11.481355    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls podName:9c912314-2d73-4cdb-91d1-f1f7eb547f53 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:19.481336854 +0000 UTC m=+51.806299124 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sn8f2" (UID: "9c912314-2d73-4cdb-91d1-f1f7eb547f53") : secret "insights-runtime-extractor-tls" not found
Jul 09 09:34:11.482297 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:11.482274    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9c912314-2d73-4cdb-91d1-f1f7eb547f53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:18.840947 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:18.840908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:18.844188 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:18.844165    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls\") pod \"image-registry-6ccb779584-t52jv\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") " pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:18.917309 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:18.917277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:18.941343 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:18.941314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:18.943940 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:18.943914    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ffe7afe-800e-4a88-a39d-2b9bd0433a87-metrics-tls\") pod \"dns-default-fpvfk\" (UID: \"9ffe7afe-800e-4a88-a39d-2b9bd0433a87\") " pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:18.962021 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:18.961993    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:19.067143 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.067112    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6ccb779584-t52jv"]
Jul 09 09:34:19.070100 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:19.070070    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0a3548d_6a6b_45c6_ae75_94c83e693f7c.slice/crio-c129626ab700ab7d617fcead153e9065a87ab8e1ce24a03163ee9d3f6ed0b9c2 WatchSource:0}: Error finding container c129626ab700ab7d617fcead153e9065a87ab8e1ce24a03163ee9d3f6ed0b9c2: Status 404 returned error can't find the container with id c129626ab700ab7d617fcead153e9065a87ab8e1ce24a03163ee9d3f6ed0b9c2
Jul 09 09:34:19.110032 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.109992    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fpvfk"]
Jul 09 09:34:19.112157 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:19.112134    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffe7afe_800e_4a88_a39d_2b9bd0433a87.slice/crio-f7548db7e94be41376e4eab9caab516cb01eb3ed36ac5d2fcec799dacff7bc2a WatchSource:0}: Error finding container f7548db7e94be41376e4eab9caab516cb01eb3ed36ac5d2fcec799dacff7bc2a: Status 404 returned error can't find the container with id f7548db7e94be41376e4eab9caab516cb01eb3ed36ac5d2fcec799dacff7bc2a
Jul 09 09:34:19.473347 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.473297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ccb779584-t52jv" event={"ID":"f0a3548d-6a6b-45c6-ae75-94c83e693f7c","Type":"ContainerStarted","Data":"688bba9ca872195da276ca675e585770ebeae410f41a13a8aef180a363afbb42"}
Jul 09 09:34:19.473514 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.473365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ccb779584-t52jv" event={"ID":"f0a3548d-6a6b-45c6-ae75-94c83e693f7c","Type":"ContainerStarted","Data":"c129626ab700ab7d617fcead153e9065a87ab8e1ce24a03163ee9d3f6ed0b9c2"}
Jul 09 09:34:19.473514 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.473393    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:19.474439 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.474418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fpvfk" event={"ID":"9ffe7afe-800e-4a88-a39d-2b9bd0433a87","Type":"ContainerStarted","Data":"f7548db7e94be41376e4eab9caab516cb01eb3ed36ac5d2fcec799dacff7bc2a"}
Jul 09 09:34:19.508438 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.508393    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6ccb779584-t52jv" podStartSLOduration=48.508376297 podStartE2EDuration="48.508376297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:19.507644483 +0000 UTC m=+51.832606769" watchObservedRunningTime="2026-07-09 09:34:19.508376297 +0000 UTC m=+51.833338617"
Jul 09 09:34:19.547637 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.547608    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:19.549799 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.549774    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9c912314-2d73-4cdb-91d1-f1f7eb547f53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sn8f2\" (UID: \"9c912314-2d73-4cdb-91d1-f1f7eb547f53\") " pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:19.637043 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.637004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sn8f2"
Jul 09 09:34:19.812350 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:19.812308    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sn8f2"]
Jul 09 09:34:19.817593 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:19.817563    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c912314_2d73_4cdb_91d1_f1f7eb547f53.slice/crio-d508916599a5c8c010db3ea0f417a9fe97b389d706fdccd2b787956165dcd2ca WatchSource:0}: Error finding container d508916599a5c8c010db3ea0f417a9fe97b389d706fdccd2b787956165dcd2ca: Status 404 returned error can't find the container with id d508916599a5c8c010db3ea0f417a9fe97b389d706fdccd2b787956165dcd2ca
Jul 09 09:34:20.480632 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:20.480556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sn8f2" event={"ID":"9c912314-2d73-4cdb-91d1-f1f7eb547f53","Type":"ContainerStarted","Data":"d508916599a5c8c010db3ea0f417a9fe97b389d706fdccd2b787956165dcd2ca"}
Jul 09 09:34:21.486830 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:21.486796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fpvfk" event={"ID":"9ffe7afe-800e-4a88-a39d-2b9bd0433a87","Type":"ContainerStarted","Data":"b7c34f62befef21b1fd995985c13c32d5ed12250de3d82c981f74da8dce2b263"}
Jul 09 09:34:22.491678 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:22.491644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fpvfk" event={"ID":"9ffe7afe-800e-4a88-a39d-2b9bd0433a87","Type":"ContainerStarted","Data":"d316882dd1731bd5665a1c21518cd9d8a991808563af59349bd04dbe4caa066a"}
Jul 09 09:34:22.492118 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:22.491764    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:22.510702 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:22.510624    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fpvfk" podStartSLOduration=17.280683018 podStartE2EDuration="19.510608033s" totalImagesPullingTime="2.229925015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:22.5098453 +0000 UTC m=+54.834807597" watchObservedRunningTime="2026-07-09 09:34:22.510608033 +0000 UTC m=+54.835570317"
Jul 09 09:34:25.502822 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:25.502790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sn8f2" event={"ID":"9c912314-2d73-4cdb-91d1-f1f7eb547f53","Type":"ContainerStarted","Data":"b997aa2c93b8feebe8ca34cc4e7c93da1c96c74f54752a96b2282cf9927acc53"}
Jul 09 09:34:25.503194 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:25.502854    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sn8f2" event={"ID":"9c912314-2d73-4cdb-91d1-f1f7eb547f53","Type":"ContainerStarted","Data":"7602a50fb8c8bfb0b831329dfc37b9ec1e7e9472072f5c692ddf9afaf1568386"}
Jul 09 09:34:26.453879 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.453843    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ccb779584-t52jv"]
Jul 09 09:34:26.485188 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.485148    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"]
Jul 09 09:34:26.487247 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.487223    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"
Jul 09 09:34:26.496584 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.496559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 09:34:26.496881 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.496860    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 09:34:26.496881 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.496862    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 09:34:26.497422 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.497393    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-mxkrm\""
Jul 09 09:34:26.497536 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.497403    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 09:34:26.503901 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.503868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czprb\" (UniqueName: \"kubernetes.io/projected/038fa5c7-d3ad-404c-8c6f-045993ed5ccf-kube-api-access-czprb\") pod \"managed-serviceaccount-addon-agent-5f869967c6-kqh95\" (UID: \"038fa5c7-d3ad-404c-8c6f-045993ed5ccf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"
Jul 09 09:34:26.504223 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.503935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/038fa5c7-d3ad-404c-8c6f-045993ed5ccf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f869967c6-kqh95\" (UID: \"038fa5c7-d3ad-404c-8c6f-045993ed5ccf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"
Jul 09 09:34:26.507185 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.507168    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"]
Jul 09 09:34:26.509088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.509069    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"]
Jul 09 09:34:26.509198 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.509184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:26.511342 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.511323    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.511443 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.511406    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 09:34:26.513188 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.513147    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"]
Jul 09 09:34:26.514242 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.514191    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 09:34:26.514341 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.514278    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 09:34:26.514341 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.514318    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 09:34:26.514714 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.514691    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 09:34:26.526810 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.526783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"]
Jul 09 09:34:26.531011 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.530986    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"]
Jul 09 09:34:26.568254 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.568221    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-766c776c56-ctjz7"]
Jul 09 09:34:26.570390 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.570369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.590019 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.589958    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-766c776c56-ctjz7"]
Jul 09 09:34:26.604270 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.604243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czprb\" (UniqueName: \"kubernetes.io/projected/038fa5c7-d3ad-404c-8c6f-045993ed5ccf-kube-api-access-czprb\") pod \"managed-serviceaccount-addon-agent-5f869967c6-kqh95\" (UID: \"038fa5c7-d3ad-404c-8c6f-045993ed5ccf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"
Jul 09 09:34:26.604402 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.604295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/038fa5c7-d3ad-404c-8c6f-045993ed5ccf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f869967c6-kqh95\" (UID: \"038fa5c7-d3ad-404c-8c6f-045993ed5ccf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"
Jul 09 09:34:26.607205 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.607180    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/038fa5c7-d3ad-404c-8c6f-045993ed5ccf-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5f869967c6-kqh95\" (UID: \"038fa5c7-d3ad-404c-8c6f-045993ed5ccf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"
Jul 09 09:34:26.634465 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.634445    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czprb\" (UniqueName: \"kubernetes.io/projected/038fa5c7-d3ad-404c-8c6f-045993ed5ccf-kube-api-access-czprb\") pod \"managed-serviceaccount-addon-agent-5f869967c6-kqh95\" (UID: \"038fa5c7-d3ad-404c-8c6f-045993ed5ccf\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"
Jul 09 09:34:26.705825 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.705681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aea31902-ec93-432b-a68f-99428527e1e0-tmp\") pod \"klusterlet-addon-workmgr-66d69756b9-g8p6k\" (UID: \"aea31902-ec93-432b-a68f-99428527e1e0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:26.705825 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.705732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.705825 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.705800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.706097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.705843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-ca\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.706097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.705868    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b7d7719-4327-4382-879d-6def3a0bcebb-ca-trust-extracted\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.706097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.705938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b7d7719-4327-4382-879d-6def3a0bcebb-registry-certificates\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.706097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.705985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/aea31902-ec93-432b-a68f-99428527e1e0-klusterlet-config\") pod \"klusterlet-addon-workmgr-66d69756b9-g8p6k\" (UID: \"aea31902-ec93-432b-a68f-99428527e1e0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:26.706097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.706025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bpqx\" (UniqueName: \"kubernetes.io/projected/aea31902-ec93-432b-a68f-99428527e1e0-kube-api-access-4bpqx\") pod \"klusterlet-addon-workmgr-66d69756b9-g8p6k\" (UID: \"aea31902-ec93-432b-a68f-99428527e1e0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:26.706097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.706071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b7d7719-4327-4382-879d-6def3a0bcebb-image-registry-private-configuration\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.706097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.706096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b7d7719-4327-4382-879d-6def3a0bcebb-registry-tls\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.706430 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.706158    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgpvf\" (UniqueName: \"kubernetes.io/projected/6b7d7719-4327-4382-879d-6def3a0bcebb-kube-api-access-jgpvf\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.706430 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.706202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-hub\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.706430 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.706222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b7d7719-4327-4382-879d-6def3a0bcebb-trusted-ca\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.706430 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.706238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vblck\" (UniqueName: \"kubernetes.io/projected/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-kube-api-access-vblck\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.706430 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.706277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b7d7719-4327-4382-879d-6def3a0bcebb-installation-pull-secrets\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.706430 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.706314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.706430 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.706340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b7d7719-4327-4382-879d-6def3a0bcebb-bound-sa-token\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.806149 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.806117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"
Jul 09 09:34:26.807370 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-hub\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.807512 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807380    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b7d7719-4327-4382-879d-6def3a0bcebb-trusted-ca\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.807512 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vblck\" (UniqueName: \"kubernetes.io/projected/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-kube-api-access-vblck\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.807512 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807445    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b7d7719-4327-4382-879d-6def3a0bcebb-installation-pull-secrets\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.807512 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807473    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.807512 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b7d7719-4327-4382-879d-6def3a0bcebb-bound-sa-token\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.807776 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aea31902-ec93-432b-a68f-99428527e1e0-tmp\") pod \"klusterlet-addon-workmgr-66d69756b9-g8p6k\" (UID: \"aea31902-ec93-432b-a68f-99428527e1e0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:26.807776 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.807776 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.807776 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-ca\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.807776 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807631    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b7d7719-4327-4382-879d-6def3a0bcebb-ca-trust-extracted\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.807776 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b7d7719-4327-4382-879d-6def3a0bcebb-registry-certificates\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.807776 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/aea31902-ec93-432b-a68f-99428527e1e0-klusterlet-config\") pod \"klusterlet-addon-workmgr-66d69756b9-g8p6k\" (UID: \"aea31902-ec93-432b-a68f-99428527e1e0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:26.807776 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bpqx\" (UniqueName: \"kubernetes.io/projected/aea31902-ec93-432b-a68f-99428527e1e0-kube-api-access-4bpqx\") pod \"klusterlet-addon-workmgr-66d69756b9-g8p6k\" (UID: \"aea31902-ec93-432b-a68f-99428527e1e0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:26.808333 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b7d7719-4327-4382-879d-6def3a0bcebb-image-registry-private-configuration\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.808333 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b7d7719-4327-4382-879d-6def3a0bcebb-registry-tls\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.808333 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.807965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgpvf\" (UniqueName: \"kubernetes.io/projected/6b7d7719-4327-4382-879d-6def3a0bcebb-kube-api-access-jgpvf\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.808333 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.808141    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aea31902-ec93-432b-a68f-99428527e1e0-tmp\") pod \"klusterlet-addon-workmgr-66d69756b9-g8p6k\" (UID: \"aea31902-ec93-432b-a68f-99428527e1e0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:26.808522 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.808371    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b7d7719-4327-4382-879d-6def3a0bcebb-trusted-ca\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.808575 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.808548    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b7d7719-4327-4382-879d-6def3a0bcebb-ca-trust-extracted\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.809045 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.809004    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.809126 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.809101    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b7d7719-4327-4382-879d-6def3a0bcebb-registry-certificates\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.811425 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.811376    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b7d7719-4327-4382-879d-6def3a0bcebb-image-registry-private-configuration\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.811425 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.811405    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b7d7719-4327-4382-879d-6def3a0bcebb-registry-tls\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.811425 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.811410    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.812297 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.812258    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/aea31902-ec93-432b-a68f-99428527e1e0-klusterlet-config\") pod \"klusterlet-addon-workmgr-66d69756b9-g8p6k\" (UID: \"aea31902-ec93-432b-a68f-99428527e1e0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:26.812393 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.812365    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-hub\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.812783 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.812759    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-ca\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.812994 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.812970    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.813097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.813081    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b7d7719-4327-4382-879d-6def3a0bcebb-installation-pull-secrets\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.836247 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.836212    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vblck\" (UniqueName: \"kubernetes.io/projected/8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7-kube-api-access-vblck\") pod \"cluster-proxy-proxy-agent-9c7465cfb-5xrbw\" (UID: \"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:26.836966 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.836920    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b7d7719-4327-4382-879d-6def3a0bcebb-bound-sa-token\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.837632 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.837609    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bpqx\" (UniqueName: \"kubernetes.io/projected/aea31902-ec93-432b-a68f-99428527e1e0-kube-api-access-4bpqx\") pod \"klusterlet-addon-workmgr-66d69756b9-g8p6k\" (UID: \"aea31902-ec93-432b-a68f-99428527e1e0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:26.840961 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.840908    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgpvf\" (UniqueName: \"kubernetes.io/projected/6b7d7719-4327-4382-879d-6def3a0bcebb-kube-api-access-jgpvf\") pod \"image-registry-766c776c56-ctjz7\" (UID: \"6b7d7719-4327-4382-879d-6def3a0bcebb\") " pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.897298 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.897271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:26.959454 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:26.959387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95"]
Jul 09 09:34:26.966857 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:26.966825    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod038fa5c7_d3ad_404c_8c6f_045993ed5ccf.slice/crio-32d68d1aab8b236f7584c73691d20ea86e8e8791c97f094528bda496b59eab86 WatchSource:0}: Error finding container 32d68d1aab8b236f7584c73691d20ea86e8e8791c97f094528bda496b59eab86: Status 404 returned error can't find the container with id 32d68d1aab8b236f7584c73691d20ea86e8e8791c97f094528bda496b59eab86
Jul 09 09:34:27.052556 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:27.052526    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-766c776c56-ctjz7"]
Jul 09 09:34:27.055562 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:27.055539    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7d7719_4327_4382_879d_6def3a0bcebb.slice/crio-82b4185cfd40597ddf3883dba57e6032235191416571378ca448d6ebd348d252 WatchSource:0}: Error finding container 82b4185cfd40597ddf3883dba57e6032235191416571378ca448d6ebd348d252: Status 404 returned error can't find the container with id 82b4185cfd40597ddf3883dba57e6032235191416571378ca448d6ebd348d252
Jul 09 09:34:27.122696 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:27.122671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:27.130111 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:27.130091    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"
Jul 09 09:34:27.273174 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:27.273144    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"]
Jul 09 09:34:27.297880 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:27.297692    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw"]
Jul 09 09:34:27.514968 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:27.514865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95" event={"ID":"038fa5c7-d3ad-404c-8c6f-045993ed5ccf","Type":"ContainerStarted","Data":"32d68d1aab8b236f7584c73691d20ea86e8e8791c97f094528bda496b59eab86"}
Jul 09 09:34:27.516770 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:27.516742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-766c776c56-ctjz7" event={"ID":"6b7d7719-4327-4382-879d-6def3a0bcebb","Type":"ContainerStarted","Data":"208d1f290be3193c46758fde87723fb135c96ce62b832b147256e1e426c08fd7"}
Jul 09 09:34:27.516877 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:27.516826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-766c776c56-ctjz7" event={"ID":"6b7d7719-4327-4382-879d-6def3a0bcebb","Type":"ContainerStarted","Data":"82b4185cfd40597ddf3883dba57e6032235191416571378ca448d6ebd348d252"}
Jul 09 09:34:27.516964 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:27.516936    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:27.537877 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:27.537818    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-766c776c56-ctjz7" podStartSLOduration=1.537796396 podStartE2EDuration="1.537796396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:27.535845362 +0000 UTC m=+59.860807646" watchObservedRunningTime="2026-07-09 09:34:27.537796396 +0000 UTC m=+59.862758681"
Jul 09 09:34:28.472002 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:28.471964    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea31902_ec93_432b_a68f_99428527e1e0.slice/crio-5ab735c5916eda62a0e2e4fe40187ba983b15858c533ee2486c27b4d5038c39a WatchSource:0}: Error finding container 5ab735c5916eda62a0e2e4fe40187ba983b15858c533ee2486c27b4d5038c39a: Status 404 returned error can't find the container with id 5ab735c5916eda62a0e2e4fe40187ba983b15858c533ee2486c27b4d5038c39a
Jul 09 09:34:28.473079 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:28.473056    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a8da672_bf25_46f8_8c2c_c63f6b5fd1b7.slice/crio-595c306ad7a1aaa74df62b5a8d1f1dd2cad3f7539db5da363ef068240e5751ff WatchSource:0}: Error finding container 595c306ad7a1aaa74df62b5a8d1f1dd2cad3f7539db5da363ef068240e5751ff: Status 404 returned error can't find the container with id 595c306ad7a1aaa74df62b5a8d1f1dd2cad3f7539db5da363ef068240e5751ff
Jul 09 09:34:28.520947 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:28.520911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw" event={"ID":"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7","Type":"ContainerStarted","Data":"595c306ad7a1aaa74df62b5a8d1f1dd2cad3f7539db5da363ef068240e5751ff"}
Jul 09 09:34:28.522064 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:28.522036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k" event={"ID":"aea31902-ec93-432b-a68f-99428527e1e0","Type":"ContainerStarted","Data":"5ab735c5916eda62a0e2e4fe40187ba983b15858c533ee2486c27b4d5038c39a"}
Jul 09 09:34:29.413932 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:29.413877    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vmx6b"
Jul 09 09:34:29.529106 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:29.529070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sn8f2" event={"ID":"9c912314-2d73-4cdb-91d1-f1f7eb547f53","Type":"ContainerStarted","Data":"c107f26d4b9910ef822ce91daa344af8e610c78b7102559c85e1004f32723509"}
Jul 09 09:34:29.552149 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:29.552086    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sn8f2" podStartSLOduration=17.63220988 podStartE2EDuration="26.552064666s" totalImagesPullingTime="8.919854786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:34:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:29.550866872 +0000 UTC m=+61.875829158" watchObservedRunningTime="2026-07-09 09:34:29.552064666 +0000 UTC m=+61.877026953"
Jul 09 09:34:31.539041 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:31.539005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95" event={"ID":"038fa5c7-d3ad-404c-8c6f-045993ed5ccf","Type":"ContainerStarted","Data":"048faed258809a2d5dc5fefb60033d5845a8ed7e0398941f010dcc92fa5023d0"}
Jul 09 09:34:31.556029 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:31.555979    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5f869967c6-kqh95" podStartSLOduration=1.7651360459999998 podStartE2EDuration="5.555962903s" totalImagesPullingTime="3.790826857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:31.555013261 +0000 UTC m=+63.879975546" watchObservedRunningTime="2026-07-09 09:34:31.555962903 +0000 UTC m=+63.880925187"
Jul 09 09:34:32.497417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:32.497381    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fpvfk"
Jul 09 09:34:33.553538 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.553503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k" event={"ID":"aea31902-ec93-432b-a68f-99428527e1e0","Type":"ContainerStarted","Data":"73918d24a4ebbeefcc147f0468d09e17681e7ddd538efba8990082d03df9c75b"}
Jul 09 09:34:33.553975 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.553677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:33.555527 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.555504    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k"
Jul 09 09:34:33.570560 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.570516    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-66d69756b9-g8p6k" podStartSLOduration=3.402203424 podStartE2EDuration="7.570502011s" totalImagesPullingTime="4.168298587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:33.569177804 +0000 UTC m=+65.894140089" watchObservedRunningTime="2026-07-09 09:34:33.570502011 +0000 UTC m=+65.895464273"
Jul 09 09:34:33.870592 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.870504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:34:33.873428 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.873408    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 09:34:33.883550 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.883532    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc58490a-3112-469d-b967-06c5bd8c9e20-metrics-certs\") pod \"network-metrics-daemon-5g56w\" (UID: \"cc58490a-3112-469d-b967-06c5bd8c9e20\") " pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:34:33.936061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.936036    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ntnhc\""
Jul 09 09:34:33.943947 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.943932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5g56w"
Jul 09 09:34:33.971842 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.971813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:34:33.971842 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.971851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj8bt\" (UniqueName: \"kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt\") pod \"network-check-target-9gj6d\" (UID: \"1e83b238-3f02-48c8-a05a-045a2e0fc781\") " pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:34:33.974674 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.974653    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 09:34:33.974742 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.974657    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 09:34:33.984988 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.984964    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8aa5ea73-c624-40b0-8672-e1b09f0ee72b-original-pull-secret\") pod \"global-pull-secret-syncer-wg7tt\" (UID: \"8aa5ea73-c624-40b0-8672-e1b09f0ee72b\") " pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:34:33.985938 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.985915    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 09:34:33.996814 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:33.996788    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj8bt\" (UniqueName: \"kubernetes.io/projected/1e83b238-3f02-48c8-a05a-045a2e0fc781-kube-api-access-sj8bt\") pod \"network-check-target-9gj6d\" (UID: \"1e83b238-3f02-48c8-a05a-045a2e0fc781\") " pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:34:34.080186 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:34.080154    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5g56w"]
Jul 09 09:34:34.082908 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:34.082869    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc58490a_3112_469d_b967_06c5bd8c9e20.slice/crio-fa7fc48fb7dd28cd23f8127711d62fe4e2e4974c982dac79b031ab3b38215bac WatchSource:0}: Error finding container fa7fc48fb7dd28cd23f8127711d62fe4e2e4974c982dac79b031ab3b38215bac: Status 404 returned error can't find the container with id fa7fc48fb7dd28cd23f8127711d62fe4e2e4974c982dac79b031ab3b38215bac
Jul 09 09:34:34.143154 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:34.143084    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wg7tt"
Jul 09 09:34:34.150635 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:34.150613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r7trk\""
Jul 09 09:34:34.158024 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:34.158006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:34:34.297043 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:34.296983    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wg7tt"]
Jul 09 09:34:34.299566 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:34.299539    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa5ea73_c624_40b0_8672_e1b09f0ee72b.slice/crio-dc55edeb96962f2ba7827616145251b347a8569ae479960fcc0cb4528cc3997a WatchSource:0}: Error finding container dc55edeb96962f2ba7827616145251b347a8569ae479960fcc0cb4528cc3997a: Status 404 returned error can't find the container with id dc55edeb96962f2ba7827616145251b347a8569ae479960fcc0cb4528cc3997a
Jul 09 09:34:34.317114 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:34.317093    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9gj6d"]
Jul 09 09:34:34.319164 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:34.319141    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e83b238_3f02_48c8_a05a_045a2e0fc781.slice/crio-eb8d15672c09d61cd16a6594d0480e4c1ff477d970df2d66948f39d6e8f5074e WatchSource:0}: Error finding container eb8d15672c09d61cd16a6594d0480e4c1ff477d970df2d66948f39d6e8f5074e: Status 404 returned error can't find the container with id eb8d15672c09d61cd16a6594d0480e4c1ff477d970df2d66948f39d6e8f5074e
Jul 09 09:34:34.558216 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:34.558186    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9gj6d" event={"ID":"1e83b238-3f02-48c8-a05a-045a2e0fc781","Type":"ContainerStarted","Data":"eb8d15672c09d61cd16a6594d0480e4c1ff477d970df2d66948f39d6e8f5074e"}
Jul 09 09:34:34.559454 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:34.559396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5g56w" event={"ID":"cc58490a-3112-469d-b967-06c5bd8c9e20","Type":"ContainerStarted","Data":"fa7fc48fb7dd28cd23f8127711d62fe4e2e4974c982dac79b031ab3b38215bac"}
Jul 09 09:34:34.560647 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:34.560588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wg7tt" event={"ID":"8aa5ea73-c624-40b0-8672-e1b09f0ee72b","Type":"ContainerStarted","Data":"dc55edeb96962f2ba7827616145251b347a8569ae479960fcc0cb4528cc3997a"}
Jul 09 09:34:35.567787 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:35.567522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5g56w" event={"ID":"cc58490a-3112-469d-b967-06c5bd8c9e20","Type":"ContainerStarted","Data":"23fbc29c1c65e207d491956a36cbc49142b9e652175e3dc257796b19cb73bf95"}
Jul 09 09:34:35.567787 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:35.567588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5g56w" event={"ID":"cc58490a-3112-469d-b967-06c5bd8c9e20","Type":"ContainerStarted","Data":"ae2c582e665b14b49addc2c05b5b163e8eb45271a86668be2d42d38a32688690"}
Jul 09 09:34:35.585980 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:35.585925    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5g56w" podStartSLOduration=66.677719834 podStartE2EDuration="1m7.585903636s" totalImagesPullingTime="908.183802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:35.584035142 +0000 UTC m=+67.908997438" watchObservedRunningTime="2026-07-09 09:34:35.585903636 +0000 UTC m=+67.910865921"
Jul 09 09:34:36.463091 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:36.462964    2581 patch_prober.go:28] interesting pod/image-registry-6ccb779584-t52jv container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 09:34:36.463282 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:36.463131    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6ccb779584-t52jv" podUID="f0a3548d-6a6b-45c6-ae75-94c83e693f7c" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 09:34:39.584403 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:39.584360    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wg7tt" event={"ID":"8aa5ea73-c624-40b0-8672-e1b09f0ee72b","Type":"ContainerStarted","Data":"6d907be4e91edda698d6f07e70d18ecb602c8a22c7a659e2661c226eafbd0abb"}
Jul 09 09:34:39.585921 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:39.585877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9gj6d" event={"ID":"1e83b238-3f02-48c8-a05a-045a2e0fc781","Type":"ContainerStarted","Data":"31340a255c4f9fac0fe7432dc1df7e194b17d26398eb6449a83fcdb7930ac22f"}
Jul 09 09:34:39.586041 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:39.585999    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:34:39.604401 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:39.604344    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wg7tt" podStartSLOduration=66.847979928 podStartE2EDuration="1m11.604321793s" totalImagesPullingTime="4.756341865s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:39.601743886 +0000 UTC m=+71.926706170" watchObservedRunningTime="2026-07-09 09:34:39.604321793 +0000 UTC m=+71.929284078"
Jul 09 09:34:42.599505 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:42.599465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw" event={"ID":"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7","Type":"ContainerStarted","Data":"88a9edc26f833f3fc88ad498c9b3be7bae615a7a910bbe14e2508cd70cc90059"}
Jul 09 09:34:42.599505 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:42.599507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw" event={"ID":"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7","Type":"ContainerStarted","Data":"666fc1c92d9c444d021a920eb4f4fd975c096779a82debeadce61c7b29483c6e"}
Jul 09 09:34:42.599914 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:42.599520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw" event={"ID":"8a8da672-bf25-46f8-8c2c-c63f6b5fd1b7","Type":"ContainerStarted","Data":"61cd20bccdff0979ddc0007688eac3c377e19427e1acb5de6794ec68fa40433b"}
Jul 09 09:34:42.622202 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:42.622145    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9c7465cfb-5xrbw" podStartSLOduration=3.354716884 podStartE2EDuration="16.622126987s" totalImagesPullingTime="13.267410103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:34:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:42.619853981 +0000 UTC m=+74.944816266" watchObservedRunningTime="2026-07-09 09:34:42.622126987 +0000 UTC m=+74.947089272"
Jul 09 09:34:42.622938 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:42.622874    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9gj6d" podStartSLOduration=69.888375522 podStartE2EDuration="1m14.622862907s" totalImagesPullingTime="4.734487385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:39.617587305 +0000 UTC m=+71.942549589" watchObservedRunningTime="2026-07-09 09:34:42.622862907 +0000 UTC m=+74.947825192"
Jul 09 09:34:46.459101 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.459074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:46.927568 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.927487    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-blcb8"]
Jul 09 09:34:46.930934 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.930911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:46.933055 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.933031    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 09:34:46.933176 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.933031    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 09:34:46.933249 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.933218    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 09:34:46.933779 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.933760    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 09:34:46.933914 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.933884    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 09:34:46.934009 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.933992    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gclbm\""
Jul 09 09:34:46.934159 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.934105    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 09:34:46.979761 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.979725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-wtmp\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:46.979987 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.979777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36296e6-f354-44a5-9ee7-20a9ddff8380-root\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:46.979987 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.979820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36296e6-f354-44a5-9ee7-20a9ddff8380-metrics-client-ca\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:46.979987 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.979849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:46.979987 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.979941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n97sg\" (UniqueName: \"kubernetes.io/projected/a36296e6-f354-44a5-9ee7-20a9ddff8380-kube-api-access-n97sg\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:46.979987 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.979977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-textfile\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:46.980244 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.980008    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-accelerators-collector-config\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:46.980244 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.980044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-tls\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:46.980244 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:46.980068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36296e6-f354-44a5-9ee7-20a9ddff8380-sys\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.080668 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.080630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.080668 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.080672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n97sg\" (UniqueName: \"kubernetes.io/projected/a36296e6-f354-44a5-9ee7-20a9ddff8380-kube-api-access-n97sg\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.080870 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.080701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-textfile\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.080870 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.080799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-accelerators-collector-config\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.080870 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.080843    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-tls\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.081033 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.080877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36296e6-f354-44a5-9ee7-20a9ddff8380-sys\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.081033 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.080925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-wtmp\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.081033 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.080972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36296e6-f354-44a5-9ee7-20a9ddff8380-root\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.081033 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:47.081017    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 09:34:47.081213 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.081039    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36296e6-f354-44a5-9ee7-20a9ddff8380-metrics-client-ca\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.081213 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.081072    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a36296e6-f354-44a5-9ee7-20a9ddff8380-sys\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.081213 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:34:47.081104    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-tls podName:a36296e6-f354-44a5-9ee7-20a9ddff8380 nodeName:}" failed. No retries permitted until 2026-07-09 09:34:47.581079723 +0000 UTC m=+79.906041991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-tls") pod "node-exporter-blcb8" (UID: "a36296e6-f354-44a5-9ee7-20a9ddff8380") : secret "node-exporter-tls" not found
Jul 09 09:34:47.081213 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.081186    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-textfile\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.081213 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.081207    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-wtmp\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.081442 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.081258    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a36296e6-f354-44a5-9ee7-20a9ddff8380-root\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.081442 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.081400    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-accelerators-collector-config\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.081633 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.081611    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a36296e6-f354-44a5-9ee7-20a9ddff8380-metrics-client-ca\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.083095 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.083073    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.093808 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.093781    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n97sg\" (UniqueName: \"kubernetes.io/projected/a36296e6-f354-44a5-9ee7-20a9ddff8380-kube-api-access-n97sg\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.585545 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.585506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-tls\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.587825 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.587799    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a36296e6-f354-44a5-9ee7-20a9ddff8380-node-exporter-tls\") pod \"node-exporter-blcb8\" (UID: \"a36296e6-f354-44a5-9ee7-20a9ddff8380\") " pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.841065 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:47.840984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-blcb8"
Jul 09 09:34:47.852383 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:34:47.852358    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36296e6_f354_44a5_9ee7_20a9ddff8380.slice/crio-a9c582b902e7701397cd84f121bdb3a94395b474ef8e809f0be3be80f3cef38d WatchSource:0}: Error finding container a9c582b902e7701397cd84f121bdb3a94395b474ef8e809f0be3be80f3cef38d: Status 404 returned error can't find the container with id a9c582b902e7701397cd84f121bdb3a94395b474ef8e809f0be3be80f3cef38d
Jul 09 09:34:48.526636 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:48.526608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-766c776c56-ctjz7"
Jul 09 09:34:48.622349 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:48.622317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blcb8" event={"ID":"a36296e6-f354-44a5-9ee7-20a9ddff8380","Type":"ContainerStarted","Data":"a9c582b902e7701397cd84f121bdb3a94395b474ef8e809f0be3be80f3cef38d"}
Jul 09 09:34:49.626828 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:49.626710    2581 generic.go:356] "Generic (PLEG): container finished" podID="a36296e6-f354-44a5-9ee7-20a9ddff8380" containerID="b43960a3f9229c9dc0829614f447865a6abde5e763f8f91b12b98d8a3c078f22" exitCode=0
Jul 09 09:34:49.626828 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:49.626791    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blcb8" event={"ID":"a36296e6-f354-44a5-9ee7-20a9ddff8380","Type":"ContainerDied","Data":"b43960a3f9229c9dc0829614f447865a6abde5e763f8f91b12b98d8a3c078f22"}
Jul 09 09:34:50.633285 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:50.633242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blcb8" event={"ID":"a36296e6-f354-44a5-9ee7-20a9ddff8380","Type":"ContainerStarted","Data":"694a88bc86e80582a61e85ed5314526fa0f18ae0ebf921042e13785802442ea7"}
Jul 09 09:34:50.633677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:50.633312    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-blcb8" event={"ID":"a36296e6-f354-44a5-9ee7-20a9ddff8380","Type":"ContainerStarted","Data":"630576b9d72b9e4ba85ba0f39f268e9ddba2e674a5a2fcc2a8b951078d4ade7d"}
Jul 09 09:34:50.656371 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:50.656314    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-blcb8" podStartSLOduration=3.25455389 podStartE2EDuration="4.656300768s" totalImagesPullingTime="1.339374409s" totalInitContainerRuntime="62.372469ms" isStatefulPod=true podCreationTimestamp="2026-07-09 09:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:34:50.655663736 +0000 UTC m=+82.980626021" watchObservedRunningTime="2026-07-09 09:34:50.656300768 +0000 UTC m=+82.981263052"
Jul 09 09:34:51.478950 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.478903    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6ccb779584-t52jv" podUID="f0a3548d-6a6b-45c6-ae75-94c83e693f7c" containerName="registry" containerID="cri-o://688bba9ca872195da276ca675e585770ebeae410f41a13a8aef180a363afbb42" gracePeriod=30
Jul 09 09:34:51.639115 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.639085    2581 generic.go:356] "Generic (PLEG): container finished" podID="f0a3548d-6a6b-45c6-ae75-94c83e693f7c" containerID="688bba9ca872195da276ca675e585770ebeae410f41a13a8aef180a363afbb42" exitCode=0
Jul 09 09:34:51.639476 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.639162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ccb779584-t52jv" event={"ID":"f0a3548d-6a6b-45c6-ae75-94c83e693f7c","Type":"ContainerDied","Data":"688bba9ca872195da276ca675e585770ebeae410f41a13a8aef180a363afbb42"}
Jul 09 09:34:51.723698 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.723676    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:51.821472 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.821382    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-trusted-ca\") pod \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") "
Jul 09 09:34:51.821472 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.821423    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-installation-pull-secrets\") pod \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") "
Jul 09 09:34:51.821472 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.821446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhkqk\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-kube-api-access-rhkqk\") pod \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") "
Jul 09 09:34:51.821748 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.821487    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls\") pod \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") "
Jul 09 09:34:51.821748 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.821507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-image-registry-private-configuration\") pod \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") "
Jul 09 09:34:51.821748 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.821526    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-certificates\") pod \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") "
Jul 09 09:34:51.821748 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.821551    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-bound-sa-token\") pod \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") "
Jul 09 09:34:51.821748 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.821583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-ca-trust-extracted\") pod \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\" (UID: \"f0a3548d-6a6b-45c6-ae75-94c83e693f7c\") "
Jul 09 09:34:51.822063 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.821822    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f0a3548d-6a6b-45c6-ae75-94c83e693f7c" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:34:51.822297 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.822252    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f0a3548d-6a6b-45c6-ae75-94c83e693f7c" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:34:51.824177 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.824150    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f0a3548d-6a6b-45c6-ae75-94c83e693f7c" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:34:51.824345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.824308    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-kube-api-access-rhkqk" (OuterVolumeSpecName: "kube-api-access-rhkqk") pod "f0a3548d-6a6b-45c6-ae75-94c83e693f7c" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c"). InnerVolumeSpecName "kube-api-access-rhkqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:34:51.824406 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.824351    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f0a3548d-6a6b-45c6-ae75-94c83e693f7c" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:34:51.824446 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.824417    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f0a3548d-6a6b-45c6-ae75-94c83e693f7c" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:34:51.824481 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.824441    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f0a3548d-6a6b-45c6-ae75-94c83e693f7c" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:34:51.838332 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.838307    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f0a3548d-6a6b-45c6-ae75-94c83e693f7c" (UID: "f0a3548d-6a6b-45c6-ae75-94c83e693f7c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:34:51.922881 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.922845    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-tls\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:34:51.922881 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.922874    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-image-registry-private-configuration\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:34:51.923096 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.922903    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-registry-certificates\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:34:51.923096 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.922918    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-bound-sa-token\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:34:51.923096 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.922930    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-ca-trust-extracted\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:34:51.923096 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.922941    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-trusted-ca\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:34:51.923096 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.922951    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-installation-pull-secrets\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:34:51.923096 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:51.922962    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhkqk\" (UniqueName: \"kubernetes.io/projected/f0a3548d-6a6b-45c6-ae75-94c83e693f7c-kube-api-access-rhkqk\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:34:52.643361 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:52.643274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6ccb779584-t52jv" event={"ID":"f0a3548d-6a6b-45c6-ae75-94c83e693f7c","Type":"ContainerDied","Data":"c129626ab700ab7d617fcead153e9065a87ab8e1ce24a03163ee9d3f6ed0b9c2"}
Jul 09 09:34:52.643361 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:52.643307    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6ccb779584-t52jv"
Jul 09 09:34:52.643361 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:52.643336    2581 scope.go:117] "RemoveContainer" containerID="688bba9ca872195da276ca675e585770ebeae410f41a13a8aef180a363afbb42"
Jul 09 09:34:52.659972 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:52.659940    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6ccb779584-t52jv"]
Jul 09 09:34:52.663270 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:52.663246    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6ccb779584-t52jv"]
Jul 09 09:34:54.227256 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:34:54.227228    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0a3548d-6a6b-45c6-ae75-94c83e693f7c" path="/var/lib/kubelet/pods/f0a3548d-6a6b-45c6-ae75-94c83e693f7c/volumes"
Jul 09 09:35:10.593188 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:35:10.593158    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9gj6d"
Jul 09 09:35:11.342359 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:35:11.342330    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blcb8_a36296e6-f354-44a5-9ee7-20a9ddff8380/init-textfile/0.log"
Jul 09 09:35:11.541502 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:35:11.541475    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blcb8_a36296e6-f354-44a5-9ee7-20a9ddff8380/node-exporter/0.log"
Jul 09 09:35:11.740217 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:35:11.740195    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blcb8_a36296e6-f354-44a5-9ee7-20a9ddff8380/kube-rbac-proxy/0.log"
Jul 09 09:38:28.044574 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:38:28.044540    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:38:28.048922 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:38:28.048899    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:38:28.049546 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:38:28.049523    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 09:41:43.993687 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:43.993650    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w4wkf"]
Jul 09 09:41:43.994301 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:43.993981    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a3548d-6a6b-45c6-ae75-94c83e693f7c" containerName="registry"
Jul 09 09:41:43.994301 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:43.993999    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a3548d-6a6b-45c6-ae75-94c83e693f7c" containerName="registry"
Jul 09 09:41:43.994301 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:43.994085    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a3548d-6a6b-45c6-ae75-94c83e693f7c" containerName="registry"
Jul 09 09:41:43.995860 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:43.995841    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-w4wkf"
Jul 09 09:41:43.998278 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:43.998245    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 09 09:41:43.998399 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:43.998375    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 09 09:41:43.998474 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:43.998456    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vzzvv\""
Jul 09 09:41:44.007261 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.007242    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w4wkf"]
Jul 09 09:41:44.109629 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.109587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ccf7aa-a6ce-4411-be0e-416fae63c807-bound-sa-token\") pod \"cert-manager-55b4d954d5-w4wkf\" (UID: \"27ccf7aa-a6ce-4411-be0e-416fae63c807\") " pod="cert-manager/cert-manager-55b4d954d5-w4wkf"
Jul 09 09:41:44.109629 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.109629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtgm8\" (UniqueName: \"kubernetes.io/projected/27ccf7aa-a6ce-4411-be0e-416fae63c807-kube-api-access-gtgm8\") pod \"cert-manager-55b4d954d5-w4wkf\" (UID: \"27ccf7aa-a6ce-4411-be0e-416fae63c807\") " pod="cert-manager/cert-manager-55b4d954d5-w4wkf"
Jul 09 09:41:44.210571 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.210533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ccf7aa-a6ce-4411-be0e-416fae63c807-bound-sa-token\") pod \"cert-manager-55b4d954d5-w4wkf\" (UID: \"27ccf7aa-a6ce-4411-be0e-416fae63c807\") " pod="cert-manager/cert-manager-55b4d954d5-w4wkf"
Jul 09 09:41:44.210718 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.210586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtgm8\" (UniqueName: \"kubernetes.io/projected/27ccf7aa-a6ce-4411-be0e-416fae63c807-kube-api-access-gtgm8\") pod \"cert-manager-55b4d954d5-w4wkf\" (UID: \"27ccf7aa-a6ce-4411-be0e-416fae63c807\") " pod="cert-manager/cert-manager-55b4d954d5-w4wkf"
Jul 09 09:41:44.218660 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.218631    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/27ccf7aa-a6ce-4411-be0e-416fae63c807-bound-sa-token\") pod \"cert-manager-55b4d954d5-w4wkf\" (UID: \"27ccf7aa-a6ce-4411-be0e-416fae63c807\") " pod="cert-manager/cert-manager-55b4d954d5-w4wkf"
Jul 09 09:41:44.218933 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.218913    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtgm8\" (UniqueName: \"kubernetes.io/projected/27ccf7aa-a6ce-4411-be0e-416fae63c807-kube-api-access-gtgm8\") pod \"cert-manager-55b4d954d5-w4wkf\" (UID: \"27ccf7aa-a6ce-4411-be0e-416fae63c807\") " pod="cert-manager/cert-manager-55b4d954d5-w4wkf"
Jul 09 09:41:44.304647 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.304553    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-w4wkf"
Jul 09 09:41:44.424943 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.424912    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-w4wkf"]
Jul 09 09:41:44.427723 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:41:44.427696    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27ccf7aa_a6ce_4411_be0e_416fae63c807.slice/crio-96d25c118820c9d76c2b2b197e34c7bd0b7182d0630951f872bfa671956f7830 WatchSource:0}: Error finding container 96d25c118820c9d76c2b2b197e34c7bd0b7182d0630951f872bfa671956f7830: Status 404 returned error can't find the container with id 96d25c118820c9d76c2b2b197e34c7bd0b7182d0630951f872bfa671956f7830
Jul 09 09:41:44.429454 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.429435    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 09:41:44.982960 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:44.982921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-w4wkf" event={"ID":"27ccf7aa-a6ce-4411-be0e-416fae63c807","Type":"ContainerStarted","Data":"96d25c118820c9d76c2b2b197e34c7bd0b7182d0630951f872bfa671956f7830"}
Jul 09 09:41:47.996360 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:47.996323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-w4wkf" event={"ID":"27ccf7aa-a6ce-4411-be0e-416fae63c807","Type":"ContainerStarted","Data":"efceef4d51656279c580925b2ad76e043aec061465c5f6c048e306f0d5515784"}
Jul 09 09:41:48.013745 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:41:48.013700    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-w4wkf" podStartSLOduration=2.209907 podStartE2EDuration="5.013684834s" totalImagesPullingTime="2.803777834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:41:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:41:48.012516025 +0000 UTC m=+500.337478311" watchObservedRunningTime="2026-07-09 09:41:48.013684834 +0000 UTC m=+500.338647117"
Jul 09 09:42:08.255911 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.255855    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"]
Jul 09 09:42:08.260323 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.260301    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.262722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.262702    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 09 09:42:08.263733 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.263713    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 09:42:08.263846 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.263730    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 09 09:42:08.263846 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.263719    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 09:42:08.263846 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.263747    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 09 09:42:08.263846 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.263811    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zjcnv\""
Jul 09 09:42:08.269899 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.269866    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"]
Jul 09 09:42:08.386423 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.386389    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlp5t\" (UniqueName: \"kubernetes.io/projected/01877ad4-3095-4dc1-b9ab-9ba946aa691c-kube-api-access-vlp5t\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.386423 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.386423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01877ad4-3095-4dc1-b9ab-9ba946aa691c-cert\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.386638 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.386452    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/01877ad4-3095-4dc1-b9ab-9ba946aa691c-manager-config\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.386638 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.386546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01877ad4-3095-4dc1-b9ab-9ba946aa691c-metrics-cert\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.487097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.487062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01877ad4-3095-4dc1-b9ab-9ba946aa691c-metrics-cert\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.487258 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.487108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlp5t\" (UniqueName: \"kubernetes.io/projected/01877ad4-3095-4dc1-b9ab-9ba946aa691c-kube-api-access-vlp5t\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.487258 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.487125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01877ad4-3095-4dc1-b9ab-9ba946aa691c-cert\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.487258 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.487157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/01877ad4-3095-4dc1-b9ab-9ba946aa691c-manager-config\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.487853 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.487825    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/01877ad4-3095-4dc1-b9ab-9ba946aa691c-manager-config\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.489591 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.489571    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/01877ad4-3095-4dc1-b9ab-9ba946aa691c-cert\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.489650 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.489612    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/01877ad4-3095-4dc1-b9ab-9ba946aa691c-metrics-cert\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.495938 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.495910    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlp5t\" (UniqueName: \"kubernetes.io/projected/01877ad4-3095-4dc1-b9ab-9ba946aa691c-kube-api-access-vlp5t\") pod \"lws-controller-manager-599d46c4cb-m6gx7\" (UID: \"01877ad4-3095-4dc1-b9ab-9ba946aa691c\") " pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.569735 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.569635    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:08.696817 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:08.696664    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"]
Jul 09 09:42:08.699249 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:42:08.699223    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01877ad4_3095_4dc1_b9ab_9ba946aa691c.slice/crio-a9120a32380ee7f5065aac28970a35217659cf603205e2470107cb636a2f803f WatchSource:0}: Error finding container a9120a32380ee7f5065aac28970a35217659cf603205e2470107cb636a2f803f: Status 404 returned error can't find the container with id a9120a32380ee7f5065aac28970a35217659cf603205e2470107cb636a2f803f
Jul 09 09:42:09.063855 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:09.063818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7" event={"ID":"01877ad4-3095-4dc1-b9ab-9ba946aa691c","Type":"ContainerStarted","Data":"a9120a32380ee7f5065aac28970a35217659cf603205e2470107cb636a2f803f"}
Jul 09 09:42:11.073301 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:11.073207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7" event={"ID":"01877ad4-3095-4dc1-b9ab-9ba946aa691c","Type":"ContainerStarted","Data":"4289b5a6c28956126acdf2d46a6a32d9bb19a3818d4b6e58ef8230358cf0f96a"}
Jul 09 09:42:11.073736 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:11.073329    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:22.079015 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:22.078929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7"
Jul 09 09:42:22.110380 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:22.110329    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-599d46c4cb-m6gx7" podStartSLOduration=12.032258305 podStartE2EDuration="14.110316309s" totalImagesPullingTime="2.078058004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:42:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:42:11.096182783 +0000 UTC m=+523.421145066" watchObservedRunningTime="2026-07-09 09:42:22.110316309 +0000 UTC m=+534.435278593"
Jul 09 09:42:43.080351 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.080312    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"]
Jul 09 09:42:43.083614 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.083595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.088933 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.088906    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 09 09:42:43.089988 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.089832    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 09:42:43.089988 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.089864    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rjlwh\""
Jul 09 09:42:43.090134 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.090090    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 09 09:42:43.090134 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.090110    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 09:42:43.097804 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.097781    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"]
Jul 09 09:42:43.135388 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.135355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn8ld\" (UniqueName: \"kubernetes.io/projected/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-kube-api-access-bn8ld\") pod \"kuadrant-console-plugin-85fd4b8b9d-2ndhv\" (UID: \"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.135388 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.135388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-2ndhv\" (UID: \"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.135623 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.135433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-2ndhv\" (UID: \"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.236598 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.236568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-2ndhv\" (UID: \"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.236798 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.236622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn8ld\" (UniqueName: \"kubernetes.io/projected/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-kube-api-access-bn8ld\") pod \"kuadrant-console-plugin-85fd4b8b9d-2ndhv\" (UID: \"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.236798 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.236644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-2ndhv\" (UID: \"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.236798 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:42:43.236724    2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 09 09:42:43.236985 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:42:43.236806    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-plugin-serving-cert podName:033f7d4b-49b8-41aa-a7d4-d9a9a5a59565 nodeName:}" failed. No retries permitted until 2026-07-09 09:42:43.736788468 +0000 UTC m=+556.061750729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-plugin-serving-cert") pod "kuadrant-console-plugin-85fd4b8b9d-2ndhv" (UID: "033f7d4b-49b8-41aa-a7d4-d9a9a5a59565") : secret "plugin-serving-cert" not found
Jul 09 09:42:43.237238 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.237220    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-2ndhv\" (UID: \"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.247297 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.247264    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn8ld\" (UniqueName: \"kubernetes.io/projected/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-kube-api-access-bn8ld\") pod \"kuadrant-console-plugin-85fd4b8b9d-2ndhv\" (UID: \"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.741204 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.741171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-2ndhv\" (UID: \"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.743531 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.743499    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/033f7d4b-49b8-41aa-a7d4-d9a9a5a59565-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-2ndhv\" (UID: \"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:43.992249 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:43.992166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"
Jul 09 09:42:44.118544 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:44.118478    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv"]
Jul 09 09:42:44.121027 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:42:44.121002    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod033f7d4b_49b8_41aa_a7d4_d9a9a5a59565.slice/crio-2825bd7a07af337fe20061dd382aeae122ef9f80258a812c5be9096703c9af36 WatchSource:0}: Error finding container 2825bd7a07af337fe20061dd382aeae122ef9f80258a812c5be9096703c9af36: Status 404 returned error can't find the container with id 2825bd7a07af337fe20061dd382aeae122ef9f80258a812c5be9096703c9af36
Jul 09 09:42:44.185021 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:44.184990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv" event={"ID":"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565","Type":"ContainerStarted","Data":"2825bd7a07af337fe20061dd382aeae122ef9f80258a812c5be9096703c9af36"}
Jul 09 09:42:49.211860 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:49.211823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv" event={"ID":"033f7d4b-49b8-41aa-a7d4-d9a9a5a59565","Type":"ContainerStarted","Data":"5f473f0c6079195e9be8e43aee20989f8020e842839ed08391f47a7179a3665e"}
Jul 09 09:42:49.231091 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:42:49.231047    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-2ndhv" podStartSLOduration=1.657940303 podStartE2EDuration="6.231032651s" totalImagesPullingTime="4.573092348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:42:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:42:49.230135311 +0000 UTC m=+561.555097606" watchObservedRunningTime="2026-07-09 09:42:49.231032651 +0000 UTC m=+561.555994935"
Jul 09 09:43:24.611787 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.611752    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-p827n"]
Jul 09 09:43:24.629736 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.629713    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-p827n"]
Jul 09 09:43:24.629870 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.629802    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:24.632435 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.632417    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 09:43:24.715744 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.715707    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-p827n"]
Jul 09 09:43:24.749709 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.749681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnqrm\" (UniqueName: \"kubernetes.io/projected/90019966-2179-455c-84ea-c2aba20e7821-kube-api-access-wnqrm\") pod \"limitador-limitador-ff68ff5cb-p827n\" (UID: \"90019966-2179-455c-84ea-c2aba20e7821\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:24.749869 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.749722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90019966-2179-455c-84ea-c2aba20e7821-config-file\") pod \"limitador-limitador-ff68ff5cb-p827n\" (UID: \"90019966-2179-455c-84ea-c2aba20e7821\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:24.850692 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.850659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnqrm\" (UniqueName: \"kubernetes.io/projected/90019966-2179-455c-84ea-c2aba20e7821-kube-api-access-wnqrm\") pod \"limitador-limitador-ff68ff5cb-p827n\" (UID: \"90019966-2179-455c-84ea-c2aba20e7821\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:24.850853 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.850701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90019966-2179-455c-84ea-c2aba20e7821-config-file\") pod \"limitador-limitador-ff68ff5cb-p827n\" (UID: \"90019966-2179-455c-84ea-c2aba20e7821\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:24.851296 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.851277    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90019966-2179-455c-84ea-c2aba20e7821-config-file\") pod \"limitador-limitador-ff68ff5cb-p827n\" (UID: \"90019966-2179-455c-84ea-c2aba20e7821\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:24.858693 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.858670    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnqrm\" (UniqueName: \"kubernetes.io/projected/90019966-2179-455c-84ea-c2aba20e7821-kube-api-access-wnqrm\") pod \"limitador-limitador-ff68ff5cb-p827n\" (UID: \"90019966-2179-455c-84ea-c2aba20e7821\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:24.940305 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:24.940235    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:25.062920 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:25.062870    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-p827n"]
Jul 09 09:43:25.066007 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:43:25.065976    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90019966_2179_455c_84ea_c2aba20e7821.slice/crio-ae520f8a286f71f4241d11a8fb36f2036867f2cd3a0b9936e756d6d914aa4819 WatchSource:0}: Error finding container ae520f8a286f71f4241d11a8fb36f2036867f2cd3a0b9936e756d6d914aa4819: Status 404 returned error can't find the container with id ae520f8a286f71f4241d11a8fb36f2036867f2cd3a0b9936e756d6d914aa4819
Jul 09 09:43:25.331064 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:25.331031    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n" event={"ID":"90019966-2179-455c-84ea-c2aba20e7821","Type":"ContainerStarted","Data":"ae520f8a286f71f4241d11a8fb36f2036867f2cd3a0b9936e756d6d914aa4819"}
Jul 09 09:43:25.637721 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:25.637640    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-jcb57"]
Jul 09 09:43:25.641937 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:25.641922    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-jcb57"
Jul 09 09:43:25.644635 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:25.644617    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sj47w\""
Jul 09 09:43:25.649138 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:25.649118    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-jcb57"]
Jul 09 09:43:25.759487 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:25.759444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stftt\" (UniqueName: \"kubernetes.io/projected/6c944865-ab04-4b40-bfda-3add45a76fe7-kube-api-access-stftt\") pod \"authorino-8df697c9f-jcb57\" (UID: \"6c944865-ab04-4b40-bfda-3add45a76fe7\") " pod="kuadrant-system/authorino-8df697c9f-jcb57"
Jul 09 09:43:25.860814 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:25.860773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stftt\" (UniqueName: \"kubernetes.io/projected/6c944865-ab04-4b40-bfda-3add45a76fe7-kube-api-access-stftt\") pod \"authorino-8df697c9f-jcb57\" (UID: \"6c944865-ab04-4b40-bfda-3add45a76fe7\") " pod="kuadrant-system/authorino-8df697c9f-jcb57"
Jul 09 09:43:25.870277 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:25.870250    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stftt\" (UniqueName: \"kubernetes.io/projected/6c944865-ab04-4b40-bfda-3add45a76fe7-kube-api-access-stftt\") pod \"authorino-8df697c9f-jcb57\" (UID: \"6c944865-ab04-4b40-bfda-3add45a76fe7\") " pod="kuadrant-system/authorino-8df697c9f-jcb57"
Jul 09 09:43:25.951741 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:25.951721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-jcb57"
Jul 09 09:43:26.103021 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:26.102987    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-jcb57"]
Jul 09 09:43:26.106738 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:43:26.106709    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c944865_ab04_4b40_bfda_3add45a76fe7.slice/crio-500aa6036fa6cb02ec33f6a2df13ea0e8a5600ae81b4ffac45115053fc65d72a WatchSource:0}: Error finding container 500aa6036fa6cb02ec33f6a2df13ea0e8a5600ae81b4ffac45115053fc65d72a: Status 404 returned error can't find the container with id 500aa6036fa6cb02ec33f6a2df13ea0e8a5600ae81b4ffac45115053fc65d72a
Jul 09 09:43:26.336499 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:26.336419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-jcb57" event={"ID":"6c944865-ab04-4b40-bfda-3add45a76fe7","Type":"ContainerStarted","Data":"500aa6036fa6cb02ec33f6a2df13ea0e8a5600ae81b4ffac45115053fc65d72a"}
Jul 09 09:43:29.944353 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:29.944328    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:43:29.944730 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:29.944376    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:43:30.357762 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:30.357724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-jcb57" event={"ID":"6c944865-ab04-4b40-bfda-3add45a76fe7","Type":"ContainerStarted","Data":"c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672"}
Jul 09 09:43:30.359187 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:30.359166    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n" event={"ID":"90019966-2179-455c-84ea-c2aba20e7821","Type":"ContainerStarted","Data":"30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0"}
Jul 09 09:43:30.359302 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:30.359290    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:30.384102 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:30.384059    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-jcb57" podStartSLOduration=1.483430261 podStartE2EDuration="5.384039789s" totalImagesPullingTime="3.900609528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:43:30.383048053 +0000 UTC m=+602.708010337" watchObservedRunningTime="2026-07-09 09:43:30.384039789 +0000 UTC m=+602.709002117"
Jul 09 09:43:30.408850 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:30.408793    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n" podStartSLOduration=1.469127772 podStartE2EDuration="6.408773457s" totalImagesPullingTime="4.939645685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:43:30.406570962 +0000 UTC m=+602.731533248" watchObservedRunningTime="2026-07-09 09:43:30.408773457 +0000 UTC m=+602.733735742"
Jul 09 09:43:40.239973 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:40.239940    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-p827n"]
Jul 09 09:43:40.240380 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:40.240126    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n" podUID="90019966-2179-455c-84ea-c2aba20e7821" containerName="limitador" containerID="cri-o://30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0" gracePeriod=30
Jul 09 09:43:40.242437 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:40.242416    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:40.782308 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:40.782285    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:40.876999 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:40.876926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90019966-2179-455c-84ea-c2aba20e7821-config-file\") pod \"90019966-2179-455c-84ea-c2aba20e7821\" (UID: \"90019966-2179-455c-84ea-c2aba20e7821\") "
Jul 09 09:43:40.876999 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:40.876967    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnqrm\" (UniqueName: \"kubernetes.io/projected/90019966-2179-455c-84ea-c2aba20e7821-kube-api-access-wnqrm\") pod \"90019966-2179-455c-84ea-c2aba20e7821\" (UID: \"90019966-2179-455c-84ea-c2aba20e7821\") "
Jul 09 09:43:40.877298 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:40.877279    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90019966-2179-455c-84ea-c2aba20e7821-config-file" (OuterVolumeSpecName: "config-file") pod "90019966-2179-455c-84ea-c2aba20e7821" (UID: "90019966-2179-455c-84ea-c2aba20e7821"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:43:40.879135 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:40.879107    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90019966-2179-455c-84ea-c2aba20e7821-kube-api-access-wnqrm" (OuterVolumeSpecName: "kube-api-access-wnqrm") pod "90019966-2179-455c-84ea-c2aba20e7821" (UID: "90019966-2179-455c-84ea-c2aba20e7821"). InnerVolumeSpecName "kube-api-access-wnqrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:43:40.977755 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:40.977727    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90019966-2179-455c-84ea-c2aba20e7821-config-file\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:43:40.977755 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:40.977751    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnqrm\" (UniqueName: \"kubernetes.io/projected/90019966-2179-455c-84ea-c2aba20e7821-kube-api-access-wnqrm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:43:41.400300 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:41.400271    2581 generic.go:356] "Generic (PLEG): container finished" podID="90019966-2179-455c-84ea-c2aba20e7821" containerID="30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0" exitCode=0
Jul 09 09:43:41.400703 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:41.400319    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n"
Jul 09 09:43:41.400703 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:41.400331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n" event={"ID":"90019966-2179-455c-84ea-c2aba20e7821","Type":"ContainerDied","Data":"30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0"}
Jul 09 09:43:41.400703 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:41.400371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-p827n" event={"ID":"90019966-2179-455c-84ea-c2aba20e7821","Type":"ContainerDied","Data":"ae520f8a286f71f4241d11a8fb36f2036867f2cd3a0b9936e756d6d914aa4819"}
Jul 09 09:43:41.400703 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:41.400391    2581 scope.go:117] "RemoveContainer" containerID="30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0"
Jul 09 09:43:41.409191 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:41.409171    2581 scope.go:117] "RemoveContainer" containerID="30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0"
Jul 09 09:43:41.409441 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:43:41.409421    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0\": container with ID starting with 30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0 not found: ID does not exist" containerID="30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0"
Jul 09 09:43:41.409507 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:41.409454    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0"} err="failed to get container status \"30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0\": rpc error: code = NotFound desc = could not find container \"30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0\": container with ID starting with 30c08483861cc59e96ca4b31008bae418b4d45295b098065a8d1d5c93552c4f0 not found: ID does not exist"
Jul 09 09:43:41.438961 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:41.438937    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-p827n"]
Jul 09 09:43:41.448136 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:41.448117    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-p827n"]
Jul 09 09:43:42.227048 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:42.227019    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90019966-2179-455c-84ea-c2aba20e7821" path="/var/lib/kubelet/pods/90019966-2179-455c-84ea-c2aba20e7821/volumes"
Jul 09 09:43:51.079722 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.079645    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-z7vd9"]
Jul 09 09:43:51.080094 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.079944    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90019966-2179-455c-84ea-c2aba20e7821" containerName="limitador"
Jul 09 09:43:51.080094 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.079956    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="90019966-2179-455c-84ea-c2aba20e7821" containerName="limitador"
Jul 09 09:43:51.080094 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.080018    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="90019966-2179-455c-84ea-c2aba20e7821" containerName="limitador"
Jul 09 09:43:51.083832 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.083816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-z7vd9"
Jul 09 09:43:51.086191 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.086170    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 09 09:43:51.091199 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.091176    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-z7vd9"]
Jul 09 09:43:51.142818 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.142788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnvk\" (UniqueName: \"kubernetes.io/projected/b41afe9d-b625-4729-823d-52f537c3540c-kube-api-access-ngnvk\") pod \"authorino-7c77957575-z7vd9\" (UID: \"b41afe9d-b625-4729-823d-52f537c3540c\") " pod="kuadrant-system/authorino-7c77957575-z7vd9"
Jul 09 09:43:51.143005 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.142832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b41afe9d-b625-4729-823d-52f537c3540c-tls-cert\") pod \"authorino-7c77957575-z7vd9\" (UID: \"b41afe9d-b625-4729-823d-52f537c3540c\") " pod="kuadrant-system/authorino-7c77957575-z7vd9"
Jul 09 09:43:51.243759 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.243728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnvk\" (UniqueName: \"kubernetes.io/projected/b41afe9d-b625-4729-823d-52f537c3540c-kube-api-access-ngnvk\") pod \"authorino-7c77957575-z7vd9\" (UID: \"b41afe9d-b625-4729-823d-52f537c3540c\") " pod="kuadrant-system/authorino-7c77957575-z7vd9"
Jul 09 09:43:51.243943 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.243769    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b41afe9d-b625-4729-823d-52f537c3540c-tls-cert\") pod \"authorino-7c77957575-z7vd9\" (UID: \"b41afe9d-b625-4729-823d-52f537c3540c\") " pod="kuadrant-system/authorino-7c77957575-z7vd9"
Jul 09 09:43:51.246297 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.246274    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b41afe9d-b625-4729-823d-52f537c3540c-tls-cert\") pod \"authorino-7c77957575-z7vd9\" (UID: \"b41afe9d-b625-4729-823d-52f537c3540c\") " pod="kuadrant-system/authorino-7c77957575-z7vd9"
Jul 09 09:43:51.260283 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.260256    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnvk\" (UniqueName: \"kubernetes.io/projected/b41afe9d-b625-4729-823d-52f537c3540c-kube-api-access-ngnvk\") pod \"authorino-7c77957575-z7vd9\" (UID: \"b41afe9d-b625-4729-823d-52f537c3540c\") " pod="kuadrant-system/authorino-7c77957575-z7vd9"
Jul 09 09:43:51.392486 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.392401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-z7vd9"
Jul 09 09:43:51.523933 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:51.523904    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-z7vd9"]
Jul 09 09:43:51.527071 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:43:51.527046    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb41afe9d_b625_4729_823d_52f537c3540c.slice/crio-77b27bfb9892ba5325caba569ce607dc4d0d58317e75391527f4b72e82a45737 WatchSource:0}: Error finding container 77b27bfb9892ba5325caba569ce607dc4d0d58317e75391527f4b72e82a45737: Status 404 returned error can't find the container with id 77b27bfb9892ba5325caba569ce607dc4d0d58317e75391527f4b72e82a45737
Jul 09 09:43:52.440867 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:52.440776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-z7vd9" event={"ID":"b41afe9d-b625-4729-823d-52f537c3540c","Type":"ContainerStarted","Data":"1bf8ac564526f284886fb6886de090a678489b01bfc7287f322a786f27101ab0"}
Jul 09 09:43:52.440867 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:52.440818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-z7vd9" event={"ID":"b41afe9d-b625-4729-823d-52f537c3540c","Type":"ContainerStarted","Data":"77b27bfb9892ba5325caba569ce607dc4d0d58317e75391527f4b72e82a45737"}
Jul 09 09:43:52.458202 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:52.458152    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-z7vd9" podStartSLOduration=0.925192686 podStartE2EDuration="1.458136794s" totalImagesPullingTime="532.944108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:43:52.457163636 +0000 UTC m=+624.782125922" watchObservedRunningTime="2026-07-09 09:43:52.458136794 +0000 UTC m=+624.783099078"
Jul 09 09:43:52.487336 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:52.487302    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-jcb57"]
Jul 09 09:43:52.487557 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:52.487526    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-jcb57" podUID="6c944865-ab04-4b40-bfda-3add45a76fe7" containerName="authorino" containerID="cri-o://c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672" gracePeriod=30
Jul 09 09:43:52.738667 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:52.738647    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-jcb57"
Jul 09 09:43:52.756336 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:52.756310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-stftt\" (UniqueName: \"kubernetes.io/projected/6c944865-ab04-4b40-bfda-3add45a76fe7-kube-api-access-stftt\") pod \"6c944865-ab04-4b40-bfda-3add45a76fe7\" (UID: \"6c944865-ab04-4b40-bfda-3add45a76fe7\") "
Jul 09 09:43:52.758424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:52.758392    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c944865-ab04-4b40-bfda-3add45a76fe7-kube-api-access-stftt" (OuterVolumeSpecName: "kube-api-access-stftt") pod "6c944865-ab04-4b40-bfda-3add45a76fe7" (UID: "6c944865-ab04-4b40-bfda-3add45a76fe7"). InnerVolumeSpecName "kube-api-access-stftt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:43:52.857736 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:52.857704    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stftt\" (UniqueName: \"kubernetes.io/projected/6c944865-ab04-4b40-bfda-3add45a76fe7-kube-api-access-stftt\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:43:53.445771 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:53.445736    2581 generic.go:356] "Generic (PLEG): container finished" podID="6c944865-ab04-4b40-bfda-3add45a76fe7" containerID="c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672" exitCode=0
Jul 09 09:43:53.446273 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:53.445782    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-jcb57"
Jul 09 09:43:53.446273 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:53.445826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-jcb57" event={"ID":"6c944865-ab04-4b40-bfda-3add45a76fe7","Type":"ContainerDied","Data":"c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672"}
Jul 09 09:43:53.446273 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:53.445865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-jcb57" event={"ID":"6c944865-ab04-4b40-bfda-3add45a76fe7","Type":"ContainerDied","Data":"500aa6036fa6cb02ec33f6a2df13ea0e8a5600ae81b4ffac45115053fc65d72a"}
Jul 09 09:43:53.446273 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:53.445900    2581 scope.go:117] "RemoveContainer" containerID="c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672"
Jul 09 09:43:53.454231 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:53.454212    2581 scope.go:117] "RemoveContainer" containerID="c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672"
Jul 09 09:43:53.454502 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:43:53.454476    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672\": container with ID starting with c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672 not found: ID does not exist" containerID="c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672"
Jul 09 09:43:53.454564 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:53.454511    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672"} err="failed to get container status \"c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672\": rpc error: code = NotFound desc = could not find container \"c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672\": container with ID starting with c9bc1afd435e9d3fa27c0e574776c48d5212b818108cd3813fb8c18a83af4672 not found: ID does not exist"
Jul 09 09:43:53.471155 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:53.471135    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-jcb57"]
Jul 09 09:43:53.475466 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:53.475448    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-jcb57"]
Jul 09 09:43:54.227623 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:43:54.227588    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c944865-ab04-4b40-bfda-3add45a76fe7" path="/var/lib/kubelet/pods/6c944865-ab04-4b40-bfda-3add45a76fe7/volumes"
Jul 09 09:44:30.510423 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.510389    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-wh8l4"]
Jul 09 09:44:30.510965 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.510653    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c944865-ab04-4b40-bfda-3add45a76fe7" containerName="authorino"
Jul 09 09:44:30.510965 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.510668    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c944865-ab04-4b40-bfda-3add45a76fe7" containerName="authorino"
Jul 09 09:44:30.510965 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.510748    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c944865-ab04-4b40-bfda-3add45a76fe7" containerName="authorino"
Jul 09 09:44:30.513396 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.513375    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wh8l4"
Jul 09 09:44:30.516884 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.516866    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 09 09:44:30.517808 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.517792    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 09 09:44:30.517920 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.517822    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 09 09:44:30.517920 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.517827    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-rvmd8\""
Jul 09 09:44:30.526808 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.526786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wh8l4"]
Jul 09 09:44:30.641109 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.641073    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpg2x\" (UniqueName: \"kubernetes.io/projected/200a5fc6-cbde-42b2-8303-8f6f1971226e-kube-api-access-vpg2x\") pod \"model-serving-api-fd65d7d6-wh8l4\" (UID: \"200a5fc6-cbde-42b2-8303-8f6f1971226e\") " pod="kserve/model-serving-api-fd65d7d6-wh8l4"
Jul 09 09:44:30.641278 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.641129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200a5fc6-cbde-42b2-8303-8f6f1971226e-tls-certs\") pod \"model-serving-api-fd65d7d6-wh8l4\" (UID: \"200a5fc6-cbde-42b2-8303-8f6f1971226e\") " pod="kserve/model-serving-api-fd65d7d6-wh8l4"
Jul 09 09:44:30.741767 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.741730    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpg2x\" (UniqueName: \"kubernetes.io/projected/200a5fc6-cbde-42b2-8303-8f6f1971226e-kube-api-access-vpg2x\") pod \"model-serving-api-fd65d7d6-wh8l4\" (UID: \"200a5fc6-cbde-42b2-8303-8f6f1971226e\") " pod="kserve/model-serving-api-fd65d7d6-wh8l4"
Jul 09 09:44:30.741976 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.741784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200a5fc6-cbde-42b2-8303-8f6f1971226e-tls-certs\") pod \"model-serving-api-fd65d7d6-wh8l4\" (UID: \"200a5fc6-cbde-42b2-8303-8f6f1971226e\") " pod="kserve/model-serving-api-fd65d7d6-wh8l4"
Jul 09 09:44:30.744155 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.744128    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/200a5fc6-cbde-42b2-8303-8f6f1971226e-tls-certs\") pod \"model-serving-api-fd65d7d6-wh8l4\" (UID: \"200a5fc6-cbde-42b2-8303-8f6f1971226e\") " pod="kserve/model-serving-api-fd65d7d6-wh8l4"
Jul 09 09:44:30.753324 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.753300    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpg2x\" (UniqueName: \"kubernetes.io/projected/200a5fc6-cbde-42b2-8303-8f6f1971226e-kube-api-access-vpg2x\") pod \"model-serving-api-fd65d7d6-wh8l4\" (UID: \"200a5fc6-cbde-42b2-8303-8f6f1971226e\") " pod="kserve/model-serving-api-fd65d7d6-wh8l4"
Jul 09 09:44:30.826491 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.826412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-wh8l4"
Jul 09 09:44:30.956983 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:30.956703    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-wh8l4"]
Jul 09 09:44:31.585843 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:31.585787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wh8l4" event={"ID":"200a5fc6-cbde-42b2-8303-8f6f1971226e","Type":"ContainerStarted","Data":"2c6a5a7149719a81de64733f9639c4a86b8d49d69b98e548f7062493c9967af9"}
Jul 09 09:44:32.592255 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:32.592156    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-wh8l4" event={"ID":"200a5fc6-cbde-42b2-8303-8f6f1971226e","Type":"ContainerStarted","Data":"12e9389afc38467d175661e42ccdb1cbf0f2ac96b6dc2b1e810ac8602f6bff0a"}
Jul 09 09:44:32.592255 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:32.592237    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-wh8l4"
Jul 09 09:44:32.614092 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:32.614045    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-wh8l4" podStartSLOduration=1.250905368 podStartE2EDuration="2.61403011s" totalImagesPullingTime="1.363124742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:44:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:44:32.613178499 +0000 UTC m=+664.938140784" watchObservedRunningTime="2026-07-09 09:44:32.61403011 +0000 UTC m=+664.938992394"
Jul 09 09:44:43.600410 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:43.600383    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-wh8l4"
Jul 09 09:44:46.532519 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.532489    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-hq5mf"]
Jul 09 09:44:46.535765 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.535748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hq5mf"
Jul 09 09:44:46.537958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.537938    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 09 09:44:46.538075 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.537991    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-kq922\""
Jul 09 09:44:46.544255 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.544234    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hq5mf"]
Jul 09 09:44:46.664054 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.664016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h689d\" (UniqueName: \"kubernetes.io/projected/898f57ab-45fe-45c4-bb47-3d72b3920492-kube-api-access-h689d\") pod \"s3-init-hq5mf\" (UID: \"898f57ab-45fe-45c4-bb47-3d72b3920492\") " pod="kserve/s3-init-hq5mf"
Jul 09 09:44:46.664238 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.664084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898f57ab-45fe-45c4-bb47-3d72b3920492-model-cache\") pod \"s3-init-hq5mf\" (UID: \"898f57ab-45fe-45c4-bb47-3d72b3920492\") " pod="kserve/s3-init-hq5mf"
Jul 09 09:44:46.764522 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.764484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h689d\" (UniqueName: \"kubernetes.io/projected/898f57ab-45fe-45c4-bb47-3d72b3920492-kube-api-access-h689d\") pod \"s3-init-hq5mf\" (UID: \"898f57ab-45fe-45c4-bb47-3d72b3920492\") " pod="kserve/s3-init-hq5mf"
Jul 09 09:44:46.764700 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.764541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898f57ab-45fe-45c4-bb47-3d72b3920492-model-cache\") pod \"s3-init-hq5mf\" (UID: \"898f57ab-45fe-45c4-bb47-3d72b3920492\") " pod="kserve/s3-init-hq5mf"
Jul 09 09:44:46.764880 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.764865    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898f57ab-45fe-45c4-bb47-3d72b3920492-model-cache\") pod \"s3-init-hq5mf\" (UID: \"898f57ab-45fe-45c4-bb47-3d72b3920492\") " pod="kserve/s3-init-hq5mf"
Jul 09 09:44:46.773335 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.773314    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h689d\" (UniqueName: \"kubernetes.io/projected/898f57ab-45fe-45c4-bb47-3d72b3920492-kube-api-access-h689d\") pod \"s3-init-hq5mf\" (UID: \"898f57ab-45fe-45c4-bb47-3d72b3920492\") " pod="kserve/s3-init-hq5mf"
Jul 09 09:44:46.845556 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.845463    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hq5mf"
Jul 09 09:44:46.970744 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:46.970724    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hq5mf"]
Jul 09 09:44:46.973108 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:44:46.973082    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod898f57ab_45fe_45c4_bb47_3d72b3920492.slice/crio-8a12f9f11e219a93538148e46817922474d3ae97ceb8f616d70d3f0e556f1503 WatchSource:0}: Error finding container 8a12f9f11e219a93538148e46817922474d3ae97ceb8f616d70d3f0e556f1503: Status 404 returned error can't find the container with id 8a12f9f11e219a93538148e46817922474d3ae97ceb8f616d70d3f0e556f1503
Jul 09 09:44:47.653803 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:47.653760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hq5mf" event={"ID":"898f57ab-45fe-45c4-bb47-3d72b3920492","Type":"ContainerStarted","Data":"8a12f9f11e219a93538148e46817922474d3ae97ceb8f616d70d3f0e556f1503"}
Jul 09 09:44:50.668120 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:50.668028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hq5mf" event={"ID":"898f57ab-45fe-45c4-bb47-3d72b3920492","Type":"ContainerStarted","Data":"c7f62704fe553f172737d19c26c9370ef51aca3da0886c2061247393f4d8f7eb"}
Jul 09 09:44:52.676871 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:52.676840    2581 generic.go:356] "Generic (PLEG): container finished" podID="898f57ab-45fe-45c4-bb47-3d72b3920492" containerID="c7f62704fe553f172737d19c26c9370ef51aca3da0886c2061247393f4d8f7eb" exitCode=0
Jul 09 09:44:52.677351 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:52.676919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hq5mf" event={"ID":"898f57ab-45fe-45c4-bb47-3d72b3920492","Type":"ContainerDied","Data":"c7f62704fe553f172737d19c26c9370ef51aca3da0886c2061247393f4d8f7eb"}
Jul 09 09:44:57.700316 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:57.700227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hq5mf" event={"ID":"898f57ab-45fe-45c4-bb47-3d72b3920492","Type":"ContainerStarted","Data":"85143469017f8eeb91665ded033342e457bac0b29832fd3686d1d3cfd5d6c969"}
Jul 09 09:44:57.720860 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:44:57.720790    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-hq5mf" podStartSLOduration=2.414737437 podStartE2EDuration="11.720755922s" totalImagesPullingTime="7.967034909s" totalInitContainerRuntime="1.338983576s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:44:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:44:57.719106529 +0000 UTC m=+690.044068812" watchObservedRunningTime="2026-07-09 09:44:57.720755922 +0000 UTC m=+690.045718205"
Jul 09 09:46:02.931983 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:02.931948    2581 generic.go:356] "Generic (PLEG): container finished" podID="898f57ab-45fe-45c4-bb47-3d72b3920492" containerID="85143469017f8eeb91665ded033342e457bac0b29832fd3686d1d3cfd5d6c969" exitCode=0
Jul 09 09:46:02.932429 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:02.932009    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hq5mf" event={"ID":"898f57ab-45fe-45c4-bb47-3d72b3920492","Type":"ContainerDied","Data":"85143469017f8eeb91665ded033342e457bac0b29832fd3686d1d3cfd5d6c969"}
Jul 09 09:46:04.064255 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:04.064232    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hq5mf"
Jul 09 09:46:04.171571 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:04.171540    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898f57ab-45fe-45c4-bb47-3d72b3920492-model-cache\") pod \"898f57ab-45fe-45c4-bb47-3d72b3920492\" (UID: \"898f57ab-45fe-45c4-bb47-3d72b3920492\") "
Jul 09 09:46:04.171749 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:04.171581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h689d\" (UniqueName: \"kubernetes.io/projected/898f57ab-45fe-45c4-bb47-3d72b3920492-kube-api-access-h689d\") pod \"898f57ab-45fe-45c4-bb47-3d72b3920492\" (UID: \"898f57ab-45fe-45c4-bb47-3d72b3920492\") "
Jul 09 09:46:04.173775 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:04.173748    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898f57ab-45fe-45c4-bb47-3d72b3920492-kube-api-access-h689d" (OuterVolumeSpecName: "kube-api-access-h689d") pod "898f57ab-45fe-45c4-bb47-3d72b3920492" (UID: "898f57ab-45fe-45c4-bb47-3d72b3920492"). InnerVolumeSpecName "kube-api-access-h689d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:46:04.190467 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:04.190406    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/898f57ab-45fe-45c4-bb47-3d72b3920492-model-cache" (OuterVolumeSpecName: "model-cache") pod "898f57ab-45fe-45c4-bb47-3d72b3920492" (UID: "898f57ab-45fe-45c4-bb47-3d72b3920492"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:46:04.272017 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:04.271989    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/898f57ab-45fe-45c4-bb47-3d72b3920492-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:46:04.272017 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:04.272013    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h689d\" (UniqueName: \"kubernetes.io/projected/898f57ab-45fe-45c4-bb47-3d72b3920492-kube-api-access-h689d\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:46:04.945290 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:04.945236    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hq5mf"
Jul 09 09:46:04.945475 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:04.945248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hq5mf" event={"ID":"898f57ab-45fe-45c4-bb47-3d72b3920492","Type":"ContainerDied","Data":"8a12f9f11e219a93538148e46817922474d3ae97ceb8f616d70d3f0e556f1503"}
Jul 09 09:46:04.945475 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:04.945410    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a12f9f11e219a93538148e46817922474d3ae97ceb8f616d70d3f0e556f1503"
Jul 09 09:46:32.191188 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.191154    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"]
Jul 09 09:46:32.191771 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.191531    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898f57ab-45fe-45c4-bb47-3d72b3920492" containerName="s3-init"
Jul 09 09:46:32.191771 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.191549    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="898f57ab-45fe-45c4-bb47-3d72b3920492" containerName="s3-init"
Jul 09 09:46:32.191771 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.191576    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898f57ab-45fe-45c4-bb47-3d72b3920492" containerName="download-hf-model"
Jul 09 09:46:32.191771 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.191584    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="898f57ab-45fe-45c4-bb47-3d72b3920492" containerName="download-hf-model"
Jul 09 09:46:32.191771 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.191658    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="898f57ab-45fe-45c4-bb47-3d72b3920492" containerName="s3-init"
Jul 09 09:46:32.194305 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.194284    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.196773 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.196752    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 09 09:46:32.196906 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.196834    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkhz8\""
Jul 09 09:46:32.196975 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.196943    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 09:46:32.197128 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.197113    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 09:46:32.206784 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.206768    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"]
Jul 09 09:46:32.289408 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.289370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.289408 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.289407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.289619 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.289490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ccrl\" (UniqueName: \"kubernetes.io/projected/522edf42-fea1-42ca-af99-88cacf95682e-kube-api-access-8ccrl\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.289619 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.289556    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522edf42-fea1-42ca-af99-88cacf95682e-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.289619 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.289577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.289748 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.289631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.390357 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.390322    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.390542 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.390372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.390542 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.390394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.390542 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.390470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ccrl\" (UniqueName: \"kubernetes.io/projected/522edf42-fea1-42ca-af99-88cacf95682e-kube-api-access-8ccrl\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.390542 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.390507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522edf42-fea1-42ca-af99-88cacf95682e-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.390542 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.390532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.390800 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.390755    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.390884 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.390866    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.390972 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.390957    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.392675 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.392657    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.393052 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.393032    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522edf42-fea1-42ca-af99-88cacf95682e-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.398113 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.398084    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ccrl\" (UniqueName: \"kubernetes.io/projected/522edf42-fea1-42ca-af99-88cacf95682e-kube-api-access-8ccrl\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.503751 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.503721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:32.642857 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:32.642833    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"]
Jul 09 09:46:32.646211 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:46:32.646173    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522edf42_fea1_42ca_af99_88cacf95682e.slice/crio-6e2b86f660ced2d021f4cee11460f0680b505e7daf872a4d499984a64fb2877c WatchSource:0}: Error finding container 6e2b86f660ced2d021f4cee11460f0680b505e7daf872a4d499984a64fb2877c: Status 404 returned error can't find the container with id 6e2b86f660ced2d021f4cee11460f0680b505e7daf872a4d499984a64fb2877c
Jul 09 09:46:33.046650 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:33.046579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4" event={"ID":"522edf42-fea1-42ca-af99-88cacf95682e","Type":"ContainerStarted","Data":"6e2b86f660ced2d021f4cee11460f0680b505e7daf872a4d499984a64fb2877c"}
Jul 09 09:46:34.051954 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:34.051849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4" event={"ID":"522edf42-fea1-42ca-af99-88cacf95682e","Type":"ContainerStarted","Data":"ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00"}
Jul 09 09:46:34.075358 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:34.075291    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4" podStartSLOduration=0.963331708 podStartE2EDuration="2.075274368s" totalImagesPullingTime="1.11194266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:46:34.073607381 +0000 UTC m=+786.398569659" watchObservedRunningTime="2026-07-09 09:46:34.075274368 +0000 UTC m=+786.400236652"
Jul 09 09:46:34.504495 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:34.504452    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:42.504274 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:42.504222    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:42.516844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:42.516820    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:43.095576 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:43.095547    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:46:47.114716 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.114683    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"]
Jul 09 09:46:47.118151 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.118129    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.120608 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.120580    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 09:46:47.120854 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.120657    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 09 09:46:47.136307 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.136279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"]
Jul 09 09:46:47.213360 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.213329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-home\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.213521 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.213366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-model-cache\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.213521 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.213403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfx85\" (UniqueName: \"kubernetes.io/projected/fd3ea672-febc-49bd-aacd-2819b07b73bf-kube-api-access-bfx85\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.213521 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.213466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3ea672-febc-49bd-aacd-2819b07b73bf-tls-certs\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.213521 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.213494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.213521 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.213514    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-dshm\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.213687 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.213626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-tmp-dir\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.314609 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.314568    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-dshm\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.314785 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.314620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-tmp-dir\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.314785 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.314641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-home\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.314785 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.314663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-model-cache\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.314785 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.314691    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfx85\" (UniqueName: \"kubernetes.io/projected/fd3ea672-febc-49bd-aacd-2819b07b73bf-kube-api-access-bfx85\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.314785 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.314711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3ea672-febc-49bd-aacd-2819b07b73bf-tls-certs\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.314785 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.314748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.315233 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.315198    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-model-cache\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.315233 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.315228    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-tmp-dir\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.315379 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.315277    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.315379 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.315292    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-home\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.317011 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.316984    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-dshm\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.317294 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.317271    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3ea672-febc-49bd-aacd-2819b07b73bf-tls-certs\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.322696 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.322676    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfx85\" (UniqueName: \"kubernetes.io/projected/fd3ea672-febc-49bd-aacd-2819b07b73bf-kube-api-access-bfx85\") pod \"precise-prefix-cache-test-kserve-b49d646d8-c9pmj\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.431355 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.431283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:47.566872 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:47.566850    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"]
Jul 09 09:46:47.569235 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:46:47.569198    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3ea672_febc_49bd_aacd_2819b07b73bf.slice/crio-57d17b809ee5c56bbefa51cbe80704f9a4d86b01e971b809b72e0d0df44c552c WatchSource:0}: Error finding container 57d17b809ee5c56bbefa51cbe80704f9a4d86b01e971b809b72e0d0df44c552c: Status 404 returned error can't find the container with id 57d17b809ee5c56bbefa51cbe80704f9a4d86b01e971b809b72e0d0df44c552c
Jul 09 09:46:48.104097 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:48.104048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj" event={"ID":"fd3ea672-febc-49bd-aacd-2819b07b73bf","Type":"ContainerStarted","Data":"d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f"}
Jul 09 09:46:48.104262 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:48.104122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj" event={"ID":"fd3ea672-febc-49bd-aacd-2819b07b73bf","Type":"ContainerStarted","Data":"57d17b809ee5c56bbefa51cbe80704f9a4d86b01e971b809b72e0d0df44c552c"}
Jul 09 09:46:50.301012 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:50.300906    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"]
Jul 09 09:46:51.119263 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:51.119232    2581 generic.go:356] "Generic (PLEG): container finished" podID="fd3ea672-febc-49bd-aacd-2819b07b73bf" containerID="d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f" exitCode=0
Jul 09 09:46:51.119428 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:51.119293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj" event={"ID":"fd3ea672-febc-49bd-aacd-2819b07b73bf","Type":"ContainerDied","Data":"d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f"}
Jul 09 09:46:52.124938 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:52.124902    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj" event={"ID":"fd3ea672-febc-49bd-aacd-2819b07b73bf","Type":"ContainerStarted","Data":"8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae"}
Jul 09 09:46:52.143290 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:52.143235    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj" podStartSLOduration=1.777502971 podStartE2EDuration="5.143219719s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.365716748s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:46:52.142511799 +0000 UTC m=+804.467474094" watchObservedRunningTime="2026-07-09 09:46:52.143219719 +0000 UTC m=+804.468182026"
Jul 09 09:46:52.431782 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:52.431699    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:57.431735 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:57.431704    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:57.444383 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:57.444358    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:46:58.159647 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:46:58.159619    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:47:05.323169 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.323133    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4" podUID="522edf42-fea1-42ca-af99-88cacf95682e" containerName="main" containerID="cri-o://ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00" gracePeriod=45
Jul 09 09:47:05.567400 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.567373    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:47:05.666621 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.666539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-tmp-dir\") pod \"522edf42-fea1-42ca-af99-88cacf95682e\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") "
Jul 09 09:47:05.666621 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.666595    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-home\") pod \"522edf42-fea1-42ca-af99-88cacf95682e\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") "
Jul 09 09:47:05.666821 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.666635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-dshm\") pod \"522edf42-fea1-42ca-af99-88cacf95682e\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") "
Jul 09 09:47:05.666821 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.666690    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-model-cache\") pod \"522edf42-fea1-42ca-af99-88cacf95682e\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") "
Jul 09 09:47:05.666821 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.666740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ccrl\" (UniqueName: \"kubernetes.io/projected/522edf42-fea1-42ca-af99-88cacf95682e-kube-api-access-8ccrl\") pod \"522edf42-fea1-42ca-af99-88cacf95682e\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") "
Jul 09 09:47:05.666821 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.666767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522edf42-fea1-42ca-af99-88cacf95682e-tls-certs\") pod \"522edf42-fea1-42ca-af99-88cacf95682e\" (UID: \"522edf42-fea1-42ca-af99-88cacf95682e\") "
Jul 09 09:47:05.667063 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.666864    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "522edf42-fea1-42ca-af99-88cacf95682e" (UID: "522edf42-fea1-42ca-af99-88cacf95682e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:47:05.667063 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.666924    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-home" (OuterVolumeSpecName: "home") pod "522edf42-fea1-42ca-af99-88cacf95682e" (UID: "522edf42-fea1-42ca-af99-88cacf95682e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:47:05.667063 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.666964    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-model-cache" (OuterVolumeSpecName: "model-cache") pod "522edf42-fea1-42ca-af99-88cacf95682e" (UID: "522edf42-fea1-42ca-af99-88cacf95682e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:47:05.667186 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.667073    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:47:05.667186 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.667090    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:47:05.667186 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.667102    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:47:05.668966 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.668945    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-dshm" (OuterVolumeSpecName: "dshm") pod "522edf42-fea1-42ca-af99-88cacf95682e" (UID: "522edf42-fea1-42ca-af99-88cacf95682e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:47:05.669041 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.668985    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/522edf42-fea1-42ca-af99-88cacf95682e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "522edf42-fea1-42ca-af99-88cacf95682e" (UID: "522edf42-fea1-42ca-af99-88cacf95682e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:47:05.669041 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.669008    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/522edf42-fea1-42ca-af99-88cacf95682e-kube-api-access-8ccrl" (OuterVolumeSpecName: "kube-api-access-8ccrl") pod "522edf42-fea1-42ca-af99-88cacf95682e" (UID: "522edf42-fea1-42ca-af99-88cacf95682e"). InnerVolumeSpecName "kube-api-access-8ccrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:47:05.767627 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.767594    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ccrl\" (UniqueName: \"kubernetes.io/projected/522edf42-fea1-42ca-af99-88cacf95682e-kube-api-access-8ccrl\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:47:05.767627 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.767620    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/522edf42-fea1-42ca-af99-88cacf95682e-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:47:05.767627 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:05.767628    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/522edf42-fea1-42ca-af99-88cacf95682e-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:47:06.182018 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:06.181985    2581 generic.go:356] "Generic (PLEG): container finished" podID="522edf42-fea1-42ca-af99-88cacf95682e" containerID="ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00" exitCode=0
Jul 09 09:47:06.182182 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:06.182050    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"
Jul 09 09:47:06.182182 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:06.182072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4" event={"ID":"522edf42-fea1-42ca-af99-88cacf95682e","Type":"ContainerDied","Data":"ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00"}
Jul 09 09:47:06.182182 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:06.182147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4" event={"ID":"522edf42-fea1-42ca-af99-88cacf95682e","Type":"ContainerDied","Data":"6e2b86f660ced2d021f4cee11460f0680b505e7daf872a4d499984a64fb2877c"}
Jul 09 09:47:06.182182 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:06.182170    2581 scope.go:117] "RemoveContainer" containerID="ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00"
Jul 09 09:47:06.190431 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:06.190410    2581 scope.go:117] "RemoveContainer" containerID="ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00"
Jul 09 09:47:06.190693 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:47:06.190675    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00\": container with ID starting with ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00 not found: ID does not exist" containerID="ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00"
Jul 09 09:47:06.190755 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:06.190704    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00"} err="failed to get container status \"ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00\": rpc error: code = NotFound desc = could not find container \"ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00\": container with ID starting with ebf1c80c725c2f1ea58eecda49fbd941bb8af8f91414d2227b6fa5d611bc0f00 not found: ID does not exist"
Jul 09 09:47:06.202417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:06.202396    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"]
Jul 09 09:47:06.209312 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:06.209293    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc57wdk4"]
Jul 09 09:47:06.227465 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:06.227445    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="522edf42-fea1-42ca-af99-88cacf95682e" path="/var/lib/kubelet/pods/522edf42-fea1-42ca-af99-88cacf95682e/volumes"
Jul 09 09:47:46.709186 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.709151    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"]
Jul 09 09:47:46.709777 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.709553    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="522edf42-fea1-42ca-af99-88cacf95682e" containerName="main"
Jul 09 09:47:46.709777 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.709570    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="522edf42-fea1-42ca-af99-88cacf95682e" containerName="main"
Jul 09 09:47:46.709777 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.709643    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="522edf42-fea1-42ca-af99-88cacf95682e" containerName="main"
Jul 09 09:47:46.712460 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.712439    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.714937 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.714918    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 09 09:47:46.724277 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.724252    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"]
Jul 09 09:47:46.795805 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.795772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-model-cache\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.795805 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.795810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwj8\" (UniqueName: \"kubernetes.io/projected/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kube-api-access-9cwj8\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.796020 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.795844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-dshm\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.796020 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.795867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tmp-dir\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.796020 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.795899    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tls-certs\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.796020 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.795924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kserve-provision-location\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.796020 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.795959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-home\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.896414 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.896385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-model-cache\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.896536 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.896424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwj8\" (UniqueName: \"kubernetes.io/projected/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kube-api-access-9cwj8\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.896596 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.896550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-dshm\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.896596 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.896578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tmp-dir\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.896696 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.896605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tls-certs\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.896696 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.896627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kserve-provision-location\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.896696 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.896665    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-home\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.896843 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.896781    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-model-cache\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.896962 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.896936    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tmp-dir\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.897047 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.897022    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kserve-provision-location\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.897047 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.897031    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-home\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.898768 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.898739    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-dshm\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.899167 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.899146    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tls-certs\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:46.905221 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:46.905201    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwj8\" (UniqueName: \"kubernetes.io/projected/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kube-api-access-9cwj8\") pod \"auth-enabled-test-kserve-667d764b96-zv8h2\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:47.022727 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.022651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:47:47.058308 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.058272    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"]
Jul 09 09:47:47.060999 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.060970    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.064349 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.064326    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-pnxcg\""
Jul 09 09:47:47.098245 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.098210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.098245 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.098242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv9z4\" (UniqueName: \"kubernetes.io/projected/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kube-api-access-wv9z4\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.098477 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.098261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.098477 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.098361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.143593 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.143562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"]
Jul 09 09:47:47.184859 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.184834    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"]
Jul 09 09:47:47.185386 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:47:47.185358    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de1d85e_d1b6_4548_a09a_e1025e07fa29.slice/crio-8add1c4cd087d5c561c243c6968f74f08d7493d3e7ab101326da978ed607fe0b WatchSource:0}: Error finding container 8add1c4cd087d5c561c243c6968f74f08d7493d3e7ab101326da978ed607fe0b: Status 404 returned error can't find the container with id 8add1c4cd087d5c561c243c6968f74f08d7493d3e7ab101326da978ed607fe0b
Jul 09 09:47:47.198709 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.198689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.198779 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.198733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.198854 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.198836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv9z4\" (UniqueName: \"kubernetes.io/projected/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kube-api-access-wv9z4\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.198919 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.198875    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.199052 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.199024    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.199208 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.199190    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.201542 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.201522    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.236902 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.236866    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv9z4\" (UniqueName: \"kubernetes.io/projected/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kube-api-access-wv9z4\") pod \"auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.341075 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.340991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" event={"ID":"8de1d85e-d1b6-4548-a09a-e1025e07fa29","Type":"ContainerStarted","Data":"3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699"}
Jul 09 09:47:47.341075 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.341057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" event={"ID":"8de1d85e-d1b6-4548-a09a-e1025e07fa29","Type":"ContainerStarted","Data":"8add1c4cd087d5c561c243c6968f74f08d7493d3e7ab101326da978ed607fe0b"}
Jul 09 09:47:47.371744 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.371705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:47.510726 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:47.510700    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"]
Jul 09 09:47:47.512804 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:47:47.512778    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a01454_a9b1_4e4c_8d40_d11f5b8ecd8b.slice/crio-f52b3047d08105332aadddd95ecf5886d557ca3352d48a6160bda0dce7b86ba1 WatchSource:0}: Error finding container f52b3047d08105332aadddd95ecf5886d557ca3352d48a6160bda0dce7b86ba1: Status 404 returned error can't find the container with id f52b3047d08105332aadddd95ecf5886d557ca3352d48a6160bda0dce7b86ba1
Jul 09 09:47:48.348525 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:48.348482    2581 generic.go:356] "Generic (PLEG): container finished" podID="38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" containerID="82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52" exitCode=0
Jul 09 09:47:48.349316 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:48.348584    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98" event={"ID":"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b","Type":"ContainerDied","Data":"82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52"}
Jul 09 09:47:48.349316 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:48.348650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98" event={"ID":"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b","Type":"ContainerStarted","Data":"f52b3047d08105332aadddd95ecf5886d557ca3352d48a6160bda0dce7b86ba1"}
Jul 09 09:47:48.350216 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:48.350194    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 09:47:50.359955 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:50.359905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98" event={"ID":"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b","Type":"ContainerStarted","Data":"c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03"}
Jul 09 09:47:50.360533 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:50.360012    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:47:50.397217 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:50.397166    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98" podStartSLOduration=1.192085929 podStartE2EDuration="3.397141382s" totalImagesPullingTime="1.483015654s" totalInitContainerRuntime="722.039799ms" isStatefulPod=false podCreationTimestamp="2026-07-09 09:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:47:50.389633671 +0000 UTC m=+862.714595956" watchObservedRunningTime="2026-07-09 09:47:50.397141382 +0000 UTC m=+862.722103669"
Jul 09 09:47:51.364553 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:51.364522    2581 generic.go:356] "Generic (PLEG): container finished" podID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerID="3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699" exitCode=0
Jul 09 09:47:51.364936 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:47:51.364588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" event={"ID":"8de1d85e-d1b6-4548-a09a-e1025e07fa29","Type":"ContainerDied","Data":"3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699"}
Jul 09 09:48:10.373694 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:10.373585    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"]
Jul 09 09:48:18.499344 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:18.499307    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" event={"ID":"8de1d85e-d1b6-4548-a09a-e1025e07fa29","Type":"ContainerStarted","Data":"087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a"}
Jul 09 09:48:18.522277 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:18.522229    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podStartSLOduration=2.183095831 podStartE2EDuration="32.52220972s" totalImagesPullingTime="26.940301863s" totalInitContainerRuntime="3.398812026s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:47:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:48:18.521497169 +0000 UTC m=+890.846459466" watchObservedRunningTime="2026-07-09 09:48:18.52220972 +0000 UTC m=+890.847171996"
Jul 09 09:48:19.023442 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:19.023357    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:48:21.366758 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:21.366682    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:48:25.479660 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:25.479592    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj" podUID="fd3ea672-febc-49bd-aacd-2819b07b73bf" containerName="main" containerID="cri-o://8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae" gracePeriod=45
Jul 09 09:48:26.148922 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.148868    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj" podUID="fd3ea672-febc-49bd-aacd-2819b07b73bf" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/health\": dial tcp 10.133.0.25:8000: connect: connection refused"
Jul 09 09:48:26.301281 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.301245    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:48:26.455792 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.455761    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3ea672-febc-49bd-aacd-2819b07b73bf-tls-certs\") pod \"fd3ea672-febc-49bd-aacd-2819b07b73bf\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") "
Jul 09 09:48:26.455991 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.455815    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-home\") pod \"fd3ea672-febc-49bd-aacd-2819b07b73bf\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") "
Jul 09 09:48:26.455991 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.455847    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-dshm\") pod \"fd3ea672-febc-49bd-aacd-2819b07b73bf\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") "
Jul 09 09:48:26.455991 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.455903    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-tmp-dir\") pod \"fd3ea672-febc-49bd-aacd-2819b07b73bf\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") "
Jul 09 09:48:26.455991 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.455942    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-kserve-provision-location\") pod \"fd3ea672-febc-49bd-aacd-2819b07b73bf\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") "
Jul 09 09:48:26.456209 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.455993    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-model-cache\") pod \"fd3ea672-febc-49bd-aacd-2819b07b73bf\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") "
Jul 09 09:48:26.456209 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.456033    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfx85\" (UniqueName: \"kubernetes.io/projected/fd3ea672-febc-49bd-aacd-2819b07b73bf-kube-api-access-bfx85\") pod \"fd3ea672-febc-49bd-aacd-2819b07b73bf\" (UID: \"fd3ea672-febc-49bd-aacd-2819b07b73bf\") "
Jul 09 09:48:26.456209 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.456107    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-home" (OuterVolumeSpecName: "home") pod "fd3ea672-febc-49bd-aacd-2819b07b73bf" (UID: "fd3ea672-febc-49bd-aacd-2819b07b73bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:48:26.456372 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.456313    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:48:26.456372 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.456308    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd3ea672-febc-49bd-aacd-2819b07b73bf" (UID: "fd3ea672-febc-49bd-aacd-2819b07b73bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:48:26.456546 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.456513    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd3ea672-febc-49bd-aacd-2819b07b73bf" (UID: "fd3ea672-febc-49bd-aacd-2819b07b73bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:48:26.458374 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.458348    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3ea672-febc-49bd-aacd-2819b07b73bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd3ea672-febc-49bd-aacd-2819b07b73bf" (UID: "fd3ea672-febc-49bd-aacd-2819b07b73bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:48:26.458374 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.458357    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3ea672-febc-49bd-aacd-2819b07b73bf-kube-api-access-bfx85" (OuterVolumeSpecName: "kube-api-access-bfx85") pod "fd3ea672-febc-49bd-aacd-2819b07b73bf" (UID: "fd3ea672-febc-49bd-aacd-2819b07b73bf"). InnerVolumeSpecName "kube-api-access-bfx85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:48:26.458536 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.458436    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-dshm" (OuterVolumeSpecName: "dshm") pod "fd3ea672-febc-49bd-aacd-2819b07b73bf" (UID: "fd3ea672-febc-49bd-aacd-2819b07b73bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:48:26.510846 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.510807    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fd3ea672-febc-49bd-aacd-2819b07b73bf" (UID: "fd3ea672-febc-49bd-aacd-2819b07b73bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:48:26.531454 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.531416    2581 generic.go:356] "Generic (PLEG): container finished" podID="fd3ea672-febc-49bd-aacd-2819b07b73bf" containerID="8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae" exitCode=0
Jul 09 09:48:26.531590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.531456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj" event={"ID":"fd3ea672-febc-49bd-aacd-2819b07b73bf","Type":"ContainerDied","Data":"8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae"}
Jul 09 09:48:26.531590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.531473    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"
Jul 09 09:48:26.531590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.531511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj" event={"ID":"fd3ea672-febc-49bd-aacd-2819b07b73bf","Type":"ContainerDied","Data":"57d17b809ee5c56bbefa51cbe80704f9a4d86b01e971b809b72e0d0df44c552c"}
Jul 09 09:48:26.531590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.531564    2581 scope.go:117] "RemoveContainer" containerID="8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae"
Jul 09 09:48:26.541638 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.541618    2581 scope.go:117] "RemoveContainer" containerID="d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f"
Jul 09 09:48:26.550765 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.550737    2581 scope.go:117] "RemoveContainer" containerID="8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae"
Jul 09 09:48:26.551043 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:48:26.551022    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae\": container with ID starting with 8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae not found: ID does not exist" containerID="8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae"
Jul 09 09:48:26.551124 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.551049    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae"} err="failed to get container status \"8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae\": rpc error: code = NotFound desc = could not find container \"8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae\": container with ID starting with 8ebcdf2d54290775a61500f1a90650ef8cedd157797b91cd1cb9fdffcdd514ae not found: ID does not exist"
Jul 09 09:48:26.551124 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.551068    2581 scope.go:117] "RemoveContainer" containerID="d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f"
Jul 09 09:48:26.551310 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:48:26.551290    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f\": container with ID starting with d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f not found: ID does not exist" containerID="d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f"
Jul 09 09:48:26.551349 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.551317    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f"} err="failed to get container status \"d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f\": rpc error: code = NotFound desc = could not find container \"d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f\": container with ID starting with d1b282d02c7bbffbc02ef7b39d2cd3b1f551859fa769c5415f991d0a20de357f not found: ID does not exist"
Jul 09 09:48:26.554163 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.554144    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"]
Jul 09 09:48:26.556479 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.556457    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b49d646d8-c9pmj"]
Jul 09 09:48:26.556740 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.556721    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd3ea672-febc-49bd-aacd-2819b07b73bf-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:48:26.556796 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.556742    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:48:26.556796 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.556751    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:48:26.556796 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.556759    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:48:26.556796 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.556768    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd3ea672-febc-49bd-aacd-2819b07b73bf-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:48:26.556796 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:26.556777    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfx85\" (UniqueName: \"kubernetes.io/projected/fd3ea672-febc-49bd-aacd-2819b07b73bf-kube-api-access-bfx85\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:48:27.022941 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:27.022910    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:48:27.024621 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:27.024598    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:48:28.229797 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:28.229757    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd3ea672-febc-49bd-aacd-2819b07b73bf" path="/var/lib/kubelet/pods/fd3ea672-febc-49bd-aacd-2819b07b73bf/volumes"
Jul 09 09:48:29.973685 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:29.973660    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:48:29.974160 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:29.973944    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:48:37.023858 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:37.023810    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:48:47.024036 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:47.023985    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:48:57.024044 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:48:57.023986    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:49:07.023980 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:49:07.023938    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:49:17.023426 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:49:17.023382    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:49:27.023130 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:49:27.023083    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:49:37.024033 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:49:37.023985    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:49:47.024191 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:49:47.024139    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:49:57.024056 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:49:57.023953    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:50:07.023570 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:07.023529    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:50:17.033478 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:17.033444    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:50:17.041511 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:17.041485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:50:21.719282 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:21.719248    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"]
Jul 09 09:50:21.727169 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:21.727141    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"]
Jul 09 09:50:21.729333 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:50:21.729300    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 09:50:21.729333 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 09:50:21.729333 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 09:50:21.729333 ip-10-0-129-157 kubenswrapper[2581]:  > containerID="c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03" cmd=["/bin/sleep","15"]
Jul 09 09:50:21.729614 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:21.729350    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98" message="" err=<
Jul 09 09:50:21.729614 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 09:50:21.729614 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 09:50:21.729614 ip-10-0-129-157 kubenswrapper[2581]:  >
Jul 09 09:50:21.729614 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:50:21.729365    2581 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 09 09:50:21.729614 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 09:50:21.729614 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 09:50:21.729614 ip-10-0-129-157 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98" podUID="38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" containerName="main" containerID="cri-o://c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03"
Jul 09 09:50:21.729614 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:21.729417    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98" podUID="38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" containerName="main" containerID="cri-o://c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03" gracePeriod=60
Jul 09 09:50:21.977330 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:21.977277    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:50:22.010316 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.010283    2581 generic.go:356] "Generic (PLEG): container finished" podID="38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" containerID="c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03" exitCode=0
Jul 09 09:50:22.010481 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.010334    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"
Jul 09 09:50:22.010481 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.010349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98" event={"ID":"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b","Type":"ContainerDied","Data":"c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03"}
Jul 09 09:50:22.010481 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.010400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98" event={"ID":"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b","Type":"ContainerDied","Data":"f52b3047d08105332aadddd95ecf5886d557ca3352d48a6160bda0dce7b86ba1"}
Jul 09 09:50:22.010481 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.010428    2581 scope.go:117] "RemoveContainer" containerID="c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03"
Jul 09 09:50:22.018706 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.018687    2581 scope.go:117] "RemoveContainer" containerID="82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52"
Jul 09 09:50:22.021837 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.021815    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tokenizer-uds\") pod \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") "
Jul 09 09:50:22.021963 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.021852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tls-certs\") pod \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") "
Jul 09 09:50:22.021963 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.021876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kserve-provision-location\") pod \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") "
Jul 09 09:50:22.022087 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.021961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv9z4\" (UniqueName: \"kubernetes.io/projected/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kube-api-access-wv9z4\") pod \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\" (UID: \"38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b\") "
Jul 09 09:50:22.022159 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.022131    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" (UID: "38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:50:22.022683 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.022656    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" (UID: "38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:50:22.023972 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.023950    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kube-api-access-wv9z4" (OuterVolumeSpecName: "kube-api-access-wv9z4") pod "38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" (UID: "38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b"). InnerVolumeSpecName "kube-api-access-wv9z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:50:22.024167 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.024147    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" (UID: "38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:50:22.026655 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.026638    2581 scope.go:117] "RemoveContainer" containerID="c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03"
Jul 09 09:50:22.026938 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:50:22.026913    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03\": container with ID starting with c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03 not found: ID does not exist" containerID="c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03"
Jul 09 09:50:22.027039 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.026946    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03"} err="failed to get container status \"c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03\": rpc error: code = NotFound desc = could not find container \"c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03\": container with ID starting with c13402db704762580553c4007d8c6e8f13e7cfd9d90e74fd955fce9db6429e03 not found: ID does not exist"
Jul 09 09:50:22.027039 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.026966    2581 scope.go:117] "RemoveContainer" containerID="82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52"
Jul 09 09:50:22.027258 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:50:22.027241    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52\": container with ID starting with 82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52 not found: ID does not exist" containerID="82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52"
Jul 09 09:50:22.027299 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.027263    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52"} err="failed to get container status \"82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52\": rpc error: code = NotFound desc = could not find container \"82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52\": container with ID starting with 82822d02b95f21f7f6bc5643661c0f996c94ad80d4f60fc1eaabde6b4aa5fe52 not found: ID does not exist"
Jul 09 09:50:22.123385 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.123349    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tokenizer-uds\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:22.123385 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.123375    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:22.123385 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.123387    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:22.123385 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.123396    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv9z4\" (UniqueName: \"kubernetes.io/projected/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b-kube-api-access-wv9z4\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:22.326415 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.326385    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"]
Jul 09 09:50:22.329968 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:22.329945    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-944f8664b-vcg98"]
Jul 09 09:50:24.228171 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:24.228138    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" path="/var/lib/kubelet/pods/38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b/volumes"
Jul 09 09:50:30.085490 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085461    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"]
Jul 09 09:50:30.085844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085748    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" containerName="main"
Jul 09 09:50:30.085844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085758    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" containerName="main"
Jul 09 09:50:30.085844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085770    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" containerName="storage-initializer"
Jul 09 09:50:30.085844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085775    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" containerName="storage-initializer"
Jul 09 09:50:30.085844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085782    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3ea672-febc-49bd-aacd-2819b07b73bf" containerName="main"
Jul 09 09:50:30.085844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085788    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3ea672-febc-49bd-aacd-2819b07b73bf" containerName="main"
Jul 09 09:50:30.085844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085796    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd3ea672-febc-49bd-aacd-2819b07b73bf" containerName="storage-initializer"
Jul 09 09:50:30.085844 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085802    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd3ea672-febc-49bd-aacd-2819b07b73bf" containerName="storage-initializer"
Jul 09 09:50:30.086102 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085862    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a01454-a9b1-4e4c-8d40-d11f5b8ecd8b" containerName="main"
Jul 09 09:50:30.086102 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.085870    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd3ea672-febc-49bd-aacd-2819b07b73bf" containerName="main"
Jul 09 09:50:30.089851 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.089829    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.092205 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.092182    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 09 09:50:30.101260 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.101235    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"]
Jul 09 09:50:30.184592 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.184558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8zb\" (UniqueName: \"kubernetes.io/projected/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-kube-api-access-rk8zb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.184759 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.184602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.184759 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.184622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.184759 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.184684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.184759 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.184720    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.184916 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.184762    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.285275 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.285237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.285457 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.285299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8zb\" (UniqueName: \"kubernetes.io/projected/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-kube-api-access-rk8zb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.285457 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.285347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.285457 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.285375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.285457 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.285408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.285684 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.285459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.285737 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.285713    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.286060 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.286007    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.286060 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.286027    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.288009 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.287984    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.288148 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.288092    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.298369 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.298338    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8zb\" (UniqueName: \"kubernetes.io/projected/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-kube-api-access-rk8zb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.399143 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.399053    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:30.531835 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:30.531783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"]
Jul 09 09:50:30.534437 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:50:30.534401    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe87ba00_10f8_4e87_baf1_26bb9ec4e3fc.slice/crio-4a695f9eeacc8580ea7b19fd6e530ec94db0e3dac33b3461e76c5672edbad358 WatchSource:0}: Error finding container 4a695f9eeacc8580ea7b19fd6e530ec94db0e3dac33b3461e76c5672edbad358: Status 404 returned error can't find the container with id 4a695f9eeacc8580ea7b19fd6e530ec94db0e3dac33b3461e76c5672edbad358
Jul 09 09:50:31.050709 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:31.050675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw" event={"ID":"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc","Type":"ContainerStarted","Data":"91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b"}
Jul 09 09:50:31.050930 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:31.050737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw" event={"ID":"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc","Type":"ContainerStarted","Data":"4a695f9eeacc8580ea7b19fd6e530ec94db0e3dac33b3461e76c5672edbad358"}
Jul 09 09:50:31.070045 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:31.069996    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw" podStartSLOduration=1.069980641 podStartE2EDuration="1.069980641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:50:31.068676399 +0000 UTC m=+1023.393638684" watchObservedRunningTime="2026-07-09 09:50:31.069980641 +0000 UTC m=+1023.394942956"
Jul 09 09:50:31.399768 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:31.399686    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:36.751417 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:36.751338    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" containerID="cri-o://087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a" gracePeriod=45
Jul 09 09:50:37.034942 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.034838    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/health\": dial tcp 10.133.0.26:8000: connect: connection refused"
Jul 09 09:50:37.495351 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.495330    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:50:37.547325 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.547279    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-model-cache\") pod \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") "
Jul 09 09:50:37.547507 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.547365    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kserve-provision-location\") pod \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") "
Jul 09 09:50:37.547507 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.547406    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-home\") pod \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") "
Jul 09 09:50:37.547507 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.547426    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cwj8\" (UniqueName: \"kubernetes.io/projected/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kube-api-access-9cwj8\") pod \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") "
Jul 09 09:50:37.547507 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.547454    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tmp-dir\") pod \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") "
Jul 09 09:50:37.547507 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.547480    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tls-certs\") pod \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") "
Jul 09 09:50:37.547833 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.547552    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-dshm\") pod \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\" (UID: \"8de1d85e-d1b6-4548-a09a-e1025e07fa29\") "
Jul 09 09:50:37.547833 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.547577    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-model-cache" (OuterVolumeSpecName: "model-cache") pod "8de1d85e-d1b6-4548-a09a-e1025e07fa29" (UID: "8de1d85e-d1b6-4548-a09a-e1025e07fa29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:50:37.547833 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.547799    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:37.548068 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.548042    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-home" (OuterVolumeSpecName: "home") pod "8de1d85e-d1b6-4548-a09a-e1025e07fa29" (UID: "8de1d85e-d1b6-4548-a09a-e1025e07fa29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:50:37.550103 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.550057    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8de1d85e-d1b6-4548-a09a-e1025e07fa29" (UID: "8de1d85e-d1b6-4548-a09a-e1025e07fa29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:50:37.550215 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.550131    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-dshm" (OuterVolumeSpecName: "dshm") pod "8de1d85e-d1b6-4548-a09a-e1025e07fa29" (UID: "8de1d85e-d1b6-4548-a09a-e1025e07fa29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:50:37.550215 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.550156    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kube-api-access-9cwj8" (OuterVolumeSpecName: "kube-api-access-9cwj8") pod "8de1d85e-d1b6-4548-a09a-e1025e07fa29" (UID: "8de1d85e-d1b6-4548-a09a-e1025e07fa29"). InnerVolumeSpecName "kube-api-access-9cwj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:50:37.563551 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.563516    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8de1d85e-d1b6-4548-a09a-e1025e07fa29" (UID: "8de1d85e-d1b6-4548-a09a-e1025e07fa29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:50:37.604132 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.604088    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8de1d85e-d1b6-4548-a09a-e1025e07fa29" (UID: "8de1d85e-d1b6-4548-a09a-e1025e07fa29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:50:37.648692 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.648613    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:37.648692 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.648640    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:37.648692 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.648650    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cwj8\" (UniqueName: \"kubernetes.io/projected/8de1d85e-d1b6-4548-a09a-e1025e07fa29-kube-api-access-9cwj8\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:37.648692 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.648659    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:37.648692 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.648668    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8de1d85e-d1b6-4548-a09a-e1025e07fa29-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:37.648692 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:37.648676    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8de1d85e-d1b6-4548-a09a-e1025e07fa29-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:50:38.078182 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.078146    2581 generic.go:356] "Generic (PLEG): container finished" podID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerID="087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a" exitCode=0
Jul 09 09:50:38.078641 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.078201    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"
Jul 09 09:50:38.078641 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.078228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" event={"ID":"8de1d85e-d1b6-4548-a09a-e1025e07fa29","Type":"ContainerDied","Data":"087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a"}
Jul 09 09:50:38.078641 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.078289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2" event={"ID":"8de1d85e-d1b6-4548-a09a-e1025e07fa29","Type":"ContainerDied","Data":"8add1c4cd087d5c561c243c6968f74f08d7493d3e7ab101326da978ed607fe0b"}
Jul 09 09:50:38.078641 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.078329    2581 scope.go:117] "RemoveContainer" containerID="087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a"
Jul 09 09:50:38.087076 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.087060    2581 scope.go:117] "RemoveContainer" containerID="3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699"
Jul 09 09:50:38.094906 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.094870    2581 scope.go:117] "RemoveContainer" containerID="087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a"
Jul 09 09:50:38.095250 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:50:38.095223    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a\": container with ID starting with 087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a not found: ID does not exist" containerID="087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a"
Jul 09 09:50:38.095319 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.095250    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a"} err="failed to get container status \"087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a\": rpc error: code = NotFound desc = could not find container \"087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a\": container with ID starting with 087d0a5cec6e0dc392b5adfe9f5bc6d219a96ff7f0a119b452fd9203795c0d8a not found: ID does not exist"
Jul 09 09:50:38.095319 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.095268    2581 scope.go:117] "RemoveContainer" containerID="3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699"
Jul 09 09:50:38.095517 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:50:38.095499    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699\": container with ID starting with 3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699 not found: ID does not exist" containerID="3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699"
Jul 09 09:50:38.095555 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.095525    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699"} err="failed to get container status \"3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699\": rpc error: code = NotFound desc = could not find container \"3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699\": container with ID starting with 3f2455198eef94a821f57a6cc754f54ed6cd5a8c0bac453a66df213c09fb3699 not found: ID does not exist"
Jul 09 09:50:38.100293 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.100272    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"]
Jul 09 09:50:38.104195 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.104177    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-667d764b96-zv8h2"]
Jul 09 09:50:38.227517 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:38.227484    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" path="/var/lib/kubelet/pods/8de1d85e-d1b6-4548-a09a-e1025e07fa29/volumes"
Jul 09 09:50:40.399778 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:40.399740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:40.412265 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:40.412240    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:50:41.100608 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:50:41.100582    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:51:20.602636 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.602543    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"]
Jul 09 09:51:20.603184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.602951    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="storage-initializer"
Jul 09 09:51:20.603184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.602965    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="storage-initializer"
Jul 09 09:51:20.603184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.602977    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main"
Jul 09 09:51:20.603184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.602986    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main"
Jul 09 09:51:20.603184 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.603034    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de1d85e-d1b6-4548-a09a-e1025e07fa29" containerName="main"
Jul 09 09:51:20.606028 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.606012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.608309 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.608287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 09:51:20.608393 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.608287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 09 09:51:20.617933 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.617911    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"]
Jul 09 09:51:20.789716 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.789684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-model-cache\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.789934 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.789735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tmp-dir\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.789934 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.789759    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tls-certs\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.789934 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.789778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.789934 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.789810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-dshm\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.789934 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.789831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-home\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.789934 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.789883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-429kg\" (UniqueName: \"kubernetes.io/projected/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kube-api-access-429kg\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.836351 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.836318    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"]
Jul 09 09:51:20.839487 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.839466    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:20.841960 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.841944    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-h7wnx\""
Jul 09 09:51:20.852242 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.852216    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"]
Jul 09 09:51:20.891042 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.890973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tmp-dir\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.891042 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.891008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tls-certs\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.891042 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.891028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.891272 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.891051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-dshm\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.891272 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.891077    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-home\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.891272 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.891102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-429kg\" (UniqueName: \"kubernetes.io/projected/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kube-api-access-429kg\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.891272 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.891145    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-model-cache\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.891476 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.891420    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tmp-dir\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.891476 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.891429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.891590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.891501    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-model-cache\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.891590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.891557    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-home\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.893321 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.893298    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-dshm\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.893557 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.893543    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tls-certs\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.898774 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.898748    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-429kg\" (UniqueName: \"kubernetes.io/projected/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kube-api-access-429kg\") pod \"auth-invalid-token-test-kserve-97b67b5bd-7fwhr\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.915627 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.915609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:20.992391 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.992359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:20.992511 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.992472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:20.992574 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.992529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxdzp\" (UniqueName: \"kubernetes.io/projected/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kube-api-access-hxdzp\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:20.992632 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:20.992595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:21.045083 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.045055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"]
Jul 09 09:51:21.046793 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:51:21.046766    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40be6760_34a8_441b_aeb5_6ba7ed6f728c.slice/crio-e3ee396299fe28e16352e6f0b744a67cbac994ae055f410bc78ad0d44b4242a0 WatchSource:0}: Error finding container e3ee396299fe28e16352e6f0b744a67cbac994ae055f410bc78ad0d44b4242a0: Status 404 returned error can't find the container with id e3ee396299fe28e16352e6f0b744a67cbac994ae055f410bc78ad0d44b4242a0
Jul 09 09:51:21.093167 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.093086    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:21.093167 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.093141    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:21.093303 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.093169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxdzp\" (UniqueName: \"kubernetes.io/projected/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kube-api-access-hxdzp\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:21.093303 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.093208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:21.093554 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.093517    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:21.093554 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.093536    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:21.095762 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.095739    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:21.101582 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.101559    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxdzp\" (UniqueName: \"kubernetes.io/projected/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kube-api-access-hxdzp\") pod \"auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:21.149032 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.148945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:21.235150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.234867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" event={"ID":"40be6760-34a8-441b-aeb5-6ba7ed6f728c","Type":"ContainerStarted","Data":"91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce"}
Jul 09 09:51:21.235150 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.234945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" event={"ID":"40be6760-34a8-441b-aeb5-6ba7ed6f728c","Type":"ContainerStarted","Data":"e3ee396299fe28e16352e6f0b744a67cbac994ae055f410bc78ad0d44b4242a0"}
Jul 09 09:51:21.281450 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:21.281427    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"]
Jul 09 09:51:21.283362 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:51:21.283335    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f18ee02_18b7_41a4_8e2b_7dd6ede2ec9a.slice/crio-a0d8bbccd14b73945afb6598e406a3e188e097ded2187ce1212d989c1f42b2f3 WatchSource:0}: Error finding container a0d8bbccd14b73945afb6598e406a3e188e097ded2187ce1212d989c1f42b2f3: Status 404 returned error can't find the container with id a0d8bbccd14b73945afb6598e406a3e188e097ded2187ce1212d989c1f42b2f3
Jul 09 09:51:22.242854 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:22.242758    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" containerID="c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee" exitCode=0
Jul 09 09:51:22.243413 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:22.242840    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq" event={"ID":"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a","Type":"ContainerDied","Data":"c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee"}
Jul 09 09:51:22.243413 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:22.243013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq" event={"ID":"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a","Type":"ContainerStarted","Data":"a0d8bbccd14b73945afb6598e406a3e188e097ded2187ce1212d989c1f42b2f3"}
Jul 09 09:51:23.255391 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:23.255319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq" event={"ID":"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a","Type":"ContainerStarted","Data":"6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab"}
Jul 09 09:51:23.255868 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:23.255503    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:23.274752 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:23.274688    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq" podStartSLOduration=2.560988477 podStartE2EDuration="3.274667151s" totalImagesPullingTime="0s" totalInitContainerRuntime="713.678674ms" isStatefulPod=false podCreationTimestamp="2026-07-09 09:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:51:23.274044967 +0000 UTC m=+1075.599007252" watchObservedRunningTime="2026-07-09 09:51:23.274667151 +0000 UTC m=+1075.599629434"
Jul 09 09:51:24.260375 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:24.260350    2581 generic.go:356] "Generic (PLEG): container finished" podID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerID="91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce" exitCode=0
Jul 09 09:51:24.260648 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:24.260418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" event={"ID":"40be6760-34a8-441b-aeb5-6ba7ed6f728c","Type":"ContainerDied","Data":"91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce"}
Jul 09 09:51:25.267479 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:25.267439    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" event={"ID":"40be6760-34a8-441b-aeb5-6ba7ed6f728c","Type":"ContainerStarted","Data":"d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72"}
Jul 09 09:51:25.299354 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:25.299298    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podStartSLOduration=2.158131282 podStartE2EDuration="5.29928222s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.141150938s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:51:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:51:25.296580388 +0000 UTC m=+1077.621542677" watchObservedRunningTime="2026-07-09 09:51:25.29928222 +0000 UTC m=+1077.624244505"
Jul 09 09:51:25.915710 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:25.915674    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:30.916000 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:30.915959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:51:30.917449 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:30.917420    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:51:40.916110 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:40.916061    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:51:50.916831 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:50.916786    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:51:54.262429 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:54.262396    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:51:59.105990 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:51:59.105953    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"]
Jul 09 09:52:00.916354 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:00.916317    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:52:10.916905 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:10.916858    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:52:14.131686 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.131647    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw" podUID="fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" containerName="main" containerID="cri-o://91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b" gracePeriod=45
Jul 09 09:52:14.375787 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.375765    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:52:14.473058 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.473030    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" containerID="91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b" exitCode=0
Jul 09 09:52:14.473228 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.473093    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"
Jul 09 09:52:14.473228 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.473118    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw" event={"ID":"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc","Type":"ContainerDied","Data":"91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b"}
Jul 09 09:52:14.473228 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.473181    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw" event={"ID":"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc","Type":"ContainerDied","Data":"4a695f9eeacc8580ea7b19fd6e530ec94db0e3dac33b3461e76c5672edbad358"}
Jul 09 09:52:14.473228 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.473209    2581 scope.go:117] "RemoveContainer" containerID="91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b"
Jul 09 09:52:14.482411 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.482394    2581 scope.go:117] "RemoveContainer" containerID="91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b"
Jul 09 09:52:14.482686 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:52:14.482668    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b\": container with ID starting with 91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b not found: ID does not exist" containerID="91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b"
Jul 09 09:52:14.482739 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.482693    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b"} err="failed to get container status \"91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b\": rpc error: code = NotFound desc = could not find container \"91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b\": container with ID starting with 91e853b34f603cfd9a5ad05a16352d5008ebc831ca80d006225d15b9dbae608b not found: ID does not exist"
Jul 09 09:52:14.533787 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.533757    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tls-certs\") pod \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") "
Jul 09 09:52:14.533787 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.533793    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-dshm\") pod \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") "
Jul 09 09:52:14.534031 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.533808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-model-cache\") pod \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") "
Jul 09 09:52:14.534031 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.533839    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-home\") pod \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") "
Jul 09 09:52:14.534031 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.533883    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tmp-dir\") pod \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") "
Jul 09 09:52:14.534031 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.533934    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk8zb\" (UniqueName: \"kubernetes.io/projected/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-kube-api-access-rk8zb\") pod \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\" (UID: \"fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc\") "
Jul 09 09:52:14.534239 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.534115    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" (UID: "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:52:14.534239 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.534189    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-home" (OuterVolumeSpecName: "home") pod "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" (UID: "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:52:14.534239 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.534217    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" (UID: "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:52:14.535939 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.535912    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-kube-api-access-rk8zb" (OuterVolumeSpecName: "kube-api-access-rk8zb") pod "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" (UID: "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc"). InnerVolumeSpecName "kube-api-access-rk8zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:52:14.536041 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.535960    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-dshm" (OuterVolumeSpecName: "dshm") pod "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" (UID: "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:52:14.536041 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.535979    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" (UID: "fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:52:14.635088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.635052    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:52:14.635088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.635081    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk8zb\" (UniqueName: \"kubernetes.io/projected/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-kube-api-access-rk8zb\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:52:14.635088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.635092    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:52:14.635308 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.635105    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:52:14.635308 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.635116    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:52:14.635308 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.635125    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:52:14.797061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.797033    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"]
Jul 09 09:52:14.800543 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:14.800521    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58dd7f4cw"]
Jul 09 09:52:16.228552 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:16.228519    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" path="/var/lib/kubelet/pods/fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc/volumes"
Jul 09 09:52:20.916357 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:20.916312    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:52:30.916505 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:30.916467    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:52:40.916084 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:40.916034    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:52:50.916084 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:52:50.915988    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:53:00.916944 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:00.916878    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:53:10.916911 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:10.916845    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:53:20.926623 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:20.926595    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:53:20.934152 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:20.934128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:53:22.703597 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:22.703563    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"]
Jul 09 09:53:22.707163 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:22.707139    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"]
Jul 09 09:53:22.716229 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:53:22.716201    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 09:53:22.716229 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 09:53:22.716229 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 09:53:22.716229 ip-10-0-129-157 kubenswrapper[2581]:  > containerID="6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab" cmd=["/bin/sleep","15"]
Jul 09 09:53:22.716488 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:22.716242    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq" message="" err=<
Jul 09 09:53:22.716488 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 09:53:22.716488 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 09:53:22.716488 ip-10-0-129-157 kubenswrapper[2581]:  >
Jul 09 09:53:22.716488 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:53:22.716255    2581 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 09 09:53:22.716488 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 09:53:22.716488 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 09:53:22.716488 ip-10-0-129-157 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq" podUID="3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" containerName="main" containerID="cri-o://6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab"
Jul 09 09:53:22.716488 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:22.716296    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq" podUID="3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" containerName="main" containerID="cri-o://6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab" gracePeriod=60
Jul 09 09:53:22.960244 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:22.960193    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:53:23.072115 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.072085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxdzp\" (UniqueName: \"kubernetes.io/projected/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kube-api-access-hxdzp\") pod \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") "
Jul 09 09:53:23.072115 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.072120    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kserve-provision-location\") pod \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") "
Jul 09 09:53:23.072314 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.072156    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tls-certs\") pod \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") "
Jul 09 09:53:23.072363 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.072333    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tokenizer-uds\") pod \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\" (UID: \"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a\") "
Jul 09 09:53:23.072598 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.072572    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" (UID: "3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:53:23.072910 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.072863    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" (UID: "3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:53:23.074728 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.074709    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" (UID: "3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:53:23.074797 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.074762    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kube-api-access-hxdzp" (OuterVolumeSpecName: "kube-api-access-hxdzp") pod "3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" (UID: "3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a"). InnerVolumeSpecName "kube-api-access-hxdzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:53:23.173276 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.173245    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxdzp\" (UniqueName: \"kubernetes.io/projected/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kube-api-access-hxdzp\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:23.173276 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.173269    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:23.173276 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.173279    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:23.173486 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.173289    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a-tokenizer-uds\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:23.756904 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.756854    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" containerID="6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab" exitCode=0
Jul 09 09:53:23.757310 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.756922    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"
Jul 09 09:53:23.757310 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.756938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq" event={"ID":"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a","Type":"ContainerDied","Data":"6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab"}
Jul 09 09:53:23.757310 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.756985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq" event={"ID":"3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a","Type":"ContainerDied","Data":"a0d8bbccd14b73945afb6598e406a3e188e097ded2187ce1212d989c1f42b2f3"}
Jul 09 09:53:23.757310 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.757007    2581 scope.go:117] "RemoveContainer" containerID="6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab"
Jul 09 09:53:23.771193 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.771171    2581 scope.go:117] "RemoveContainer" containerID="c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee"
Jul 09 09:53:23.778061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.778048    2581 scope.go:117] "RemoveContainer" containerID="6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab"
Jul 09 09:53:23.778288 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:53:23.778271    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab\": container with ID starting with 6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab not found: ID does not exist" containerID="6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab"
Jul 09 09:53:23.778328 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.778296    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab"} err="failed to get container status \"6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab\": rpc error: code = NotFound desc = could not find container \"6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab\": container with ID starting with 6e74dba3e3e603da6c911fa610fd0a3672aba68fa26bea292d9803b0232980ab not found: ID does not exist"
Jul 09 09:53:23.778328 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.778312    2581 scope.go:117] "RemoveContainer" containerID="c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee"
Jul 09 09:53:23.778520 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:53:23.778502    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee\": container with ID starting with c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee not found: ID does not exist" containerID="c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee"
Jul 09 09:53:23.778555 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.778526    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee"} err="failed to get container status \"c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee\": rpc error: code = NotFound desc = could not find container \"c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee\": container with ID starting with c852c39fe47ffb735b783997e454f6e00ea3d958fd1a32d64811fd74e400d3ee not found: ID does not exist"
Jul 09 09:53:23.783897 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.783863    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"]
Jul 09 09:53:23.787378 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:23.787361    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-555b6fb47b9pqbq"]
Jul 09 09:53:24.228078 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:24.228043    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" path="/var/lib/kubelet/pods/3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a/volumes"
Jul 09 09:53:29.995993 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:29.995964    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:53:29.996899 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:29.996862    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:53:37.773221 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:37.773161    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" containerID="cri-o://d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72" gracePeriod=45
Jul 09 09:53:37.927994 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:37.927956    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.29:8000/health\": dial tcp 10.133.0.29:8000: connect: connection refused"
Jul 09 09:53:38.421065 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.421042    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:53:38.490323 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.490293    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-dshm\") pod \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") "
Jul 09 09:53:38.490502 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.490332    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tls-certs\") pod \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") "
Jul 09 09:53:38.490502 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.490347    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-home\") pod \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") "
Jul 09 09:53:38.490502 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.490379    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kserve-provision-location\") pod \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") "
Jul 09 09:53:38.490502 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.490409    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-429kg\" (UniqueName: \"kubernetes.io/projected/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kube-api-access-429kg\") pod \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") "
Jul 09 09:53:38.490502 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.490430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-model-cache\") pod \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") "
Jul 09 09:53:38.490502 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.490444    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tmp-dir\") pod \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\" (UID: \"40be6760-34a8-441b-aeb5-6ba7ed6f728c\") "
Jul 09 09:53:38.490837 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.490691    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-model-cache" (OuterVolumeSpecName: "model-cache") pod "40be6760-34a8-441b-aeb5-6ba7ed6f728c" (UID: "40be6760-34a8-441b-aeb5-6ba7ed6f728c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:53:38.491144 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.491114    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-home" (OuterVolumeSpecName: "home") pod "40be6760-34a8-441b-aeb5-6ba7ed6f728c" (UID: "40be6760-34a8-441b-aeb5-6ba7ed6f728c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:53:38.492540 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.492510    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "40be6760-34a8-441b-aeb5-6ba7ed6f728c" (UID: "40be6760-34a8-441b-aeb5-6ba7ed6f728c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:53:38.492627 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.492550    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-dshm" (OuterVolumeSpecName: "dshm") pod "40be6760-34a8-441b-aeb5-6ba7ed6f728c" (UID: "40be6760-34a8-441b-aeb5-6ba7ed6f728c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:53:38.492733 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.492712    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kube-api-access-429kg" (OuterVolumeSpecName: "kube-api-access-429kg") pod "40be6760-34a8-441b-aeb5-6ba7ed6f728c" (UID: "40be6760-34a8-441b-aeb5-6ba7ed6f728c"). InnerVolumeSpecName "kube-api-access-429kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:53:38.508424 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.508343    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "40be6760-34a8-441b-aeb5-6ba7ed6f728c" (UID: "40be6760-34a8-441b-aeb5-6ba7ed6f728c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:53:38.557827 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.557768    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "40be6760-34a8-441b-aeb5-6ba7ed6f728c" (UID: "40be6760-34a8-441b-aeb5-6ba7ed6f728c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:53:38.591788 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.591755    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:38.591788 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.591780    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:38.591788 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.591790    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:38.591788 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.591798    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:38.592080 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.591807    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-429kg\" (UniqueName: \"kubernetes.io/projected/40be6760-34a8-441b-aeb5-6ba7ed6f728c-kube-api-access-429kg\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:38.592080 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.591816    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:38.592080 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.591845    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40be6760-34a8-441b-aeb5-6ba7ed6f728c-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:53:38.810918 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.810823    2581 generic.go:356] "Generic (PLEG): container finished" podID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerID="d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72" exitCode=0
Jul 09 09:53:38.810918 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.810877    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"
Jul 09 09:53:38.811348 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.810913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" event={"ID":"40be6760-34a8-441b-aeb5-6ba7ed6f728c","Type":"ContainerDied","Data":"d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72"}
Jul 09 09:53:38.811348 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.810966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr" event={"ID":"40be6760-34a8-441b-aeb5-6ba7ed6f728c","Type":"ContainerDied","Data":"e3ee396299fe28e16352e6f0b744a67cbac994ae055f410bc78ad0d44b4242a0"}
Jul 09 09:53:38.811348 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.810990    2581 scope.go:117] "RemoveContainer" containerID="d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72"
Jul 09 09:53:38.821129 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.821109    2581 scope.go:117] "RemoveContainer" containerID="91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce"
Jul 09 09:53:38.828905 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.828869    2581 scope.go:117] "RemoveContainer" containerID="d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72"
Jul 09 09:53:38.829132 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:53:38.829113    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72\": container with ID starting with d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72 not found: ID does not exist" containerID="d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72"
Jul 09 09:53:38.829196 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.829140    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72"} err="failed to get container status \"d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72\": rpc error: code = NotFound desc = could not find container \"d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72\": container with ID starting with d0310ef8325b478acc35557842d40aa4383a650e54c273b03f2a3fe71b263f72 not found: ID does not exist"
Jul 09 09:53:38.829196 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.829157    2581 scope.go:117] "RemoveContainer" containerID="91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce"
Jul 09 09:53:38.829402 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:53:38.829384    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce\": container with ID starting with 91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce not found: ID does not exist" containerID="91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce"
Jul 09 09:53:38.829442 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.829407    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce"} err="failed to get container status \"91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce\": rpc error: code = NotFound desc = could not find container \"91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce\": container with ID starting with 91cfb3e18d48aedbb78603a186f8f06cc7a1b297e16bde487ff991b5fe4ce5ce not found: ID does not exist"
Jul 09 09:53:38.834377 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.834357    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"]
Jul 09 09:53:38.838129 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:38.838106    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-97b67b5bd-7fwhr"]
Jul 09 09:53:40.227847 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:40.227815    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" path="/var/lib/kubelet/pods/40be6760-34a8-441b-aeb5-6ba7ed6f728c/volumes"
Jul 09 09:53:45.689579 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689543    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"]
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689798    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" containerName="storage-initializer"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689809    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" containerName="storage-initializer"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689820    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" containerName="main"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689825    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" containerName="main"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689838    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689843    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689851    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="storage-initializer"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689856    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="storage-initializer"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689863    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" containerName="main"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689869    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" containerName="main"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689931    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe87ba00-10f8-4e87-baf1-26bb9ec4e3fc" containerName="main"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689940    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f18ee02-18b7-41a4-8e2b-7dd6ede2ec9a" containerName="main"
Jul 09 09:53:45.689958 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.689946    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="40be6760-34a8-441b-aeb5-6ba7ed6f728c" containerName="main"
Jul 09 09:53:45.694411 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.694393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.696853 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.696831    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 09:53:45.697615 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.697587    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 09 09:53:45.697727 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.697617    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 09:53:45.697727 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.697657    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkhz8\""
Jul 09 09:53:45.701988 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.701961    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"]
Jul 09 09:53:45.742944 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.742913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.743091 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.742953    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.743091 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.742978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t2mg\" (UniqueName: \"kubernetes.io/projected/4599ee7b-b7bf-498e-8af5-0e872bb159ad-kube-api-access-5t2mg\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.743091 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.743004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.743091 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.743062    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.743228 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.743108    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.843906 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.843859    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.844088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.843934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.844088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.843970    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.844088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.843992    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.844088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.844010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t2mg\" (UniqueName: \"kubernetes.io/projected/4599ee7b-b7bf-498e-8af5-0e872bb159ad-kube-api-access-5t2mg\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.844088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.844045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.844355 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.844294    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.844355 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.844313    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.844476 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.844459    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.846436 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.846418    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.846620 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.846601    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:45.851835 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:45.851814    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t2mg\" (UniqueName: \"kubernetes.io/projected/4599ee7b-b7bf-498e-8af5-0e872bb159ad-kube-api-access-5t2mg\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:46.005725 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.005677    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:46.114243 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.114192    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"]
Jul 09 09:53:46.117771 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.117747    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.120323 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.120303    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-hhsdw\""
Jul 09 09:53:46.127507 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.127482    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"]
Jul 09 09:53:46.140875 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.140845    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"]
Jul 09 09:53:46.143974 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:53:46.143947    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4599ee7b_b7bf_498e_8af5_0e872bb159ad.slice/crio-fa926baee1a184af60b52ef7ed5da3dff7a8fc2f9e2002b5cd2b6d21406bb73b WatchSource:0}: Error finding container fa926baee1a184af60b52ef7ed5da3dff7a8fc2f9e2002b5cd2b6d21406bb73b: Status 404 returned error can't find the container with id fa926baee1a184af60b52ef7ed5da3dff7a8fc2f9e2002b5cd2b6d21406bb73b
Jul 09 09:53:46.146304 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.146281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.146366 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.146336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szcfz\" (UniqueName: \"kubernetes.io/projected/90aac38f-36b5-49ce-a0b7-e2b8fe927906-kube-api-access-szcfz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.146425 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.146399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.247099 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.247067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szcfz\" (UniqueName: \"kubernetes.io/projected/90aac38f-36b5-49ce-a0b7-e2b8fe927906-kube-api-access-szcfz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.247283 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.247116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.247283 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.247159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.247541 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.247521    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.249493 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.249473    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.258309 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.258250    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szcfz\" (UniqueName: \"kubernetes.io/projected/90aac38f-36b5-49ce-a0b7-e2b8fe927906-kube-api-access-szcfz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.428398 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.428362    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.558146 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.558109    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"]
Jul 09 09:53:46.561875 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:53:46.561840    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90aac38f_36b5_49ce_a0b7_e2b8fe927906.slice/crio-f25a7e1d4952678a986c6f1af3779c1ecc5faeb9171ed7890a9697904d913812 WatchSource:0}: Error finding container f25a7e1d4952678a986c6f1af3779c1ecc5faeb9171ed7890a9697904d913812: Status 404 returned error can't find the container with id f25a7e1d4952678a986c6f1af3779c1ecc5faeb9171ed7890a9697904d913812
Jul 09 09:53:46.843535 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.843437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv" event={"ID":"4599ee7b-b7bf-498e-8af5-0e872bb159ad","Type":"ContainerStarted","Data":"cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4"}
Jul 09 09:53:46.843535 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.843495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv" event={"ID":"4599ee7b-b7bf-498e-8af5-0e872bb159ad","Type":"ContainerStarted","Data":"fa926baee1a184af60b52ef7ed5da3dff7a8fc2f9e2002b5cd2b6d21406bb73b"}
Jul 09 09:53:46.844834 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.844810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f" event={"ID":"90aac38f-36b5-49ce-a0b7-e2b8fe927906","Type":"ContainerStarted","Data":"3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4"}
Jul 09 09:53:46.844941 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.844853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f" event={"ID":"90aac38f-36b5-49ce-a0b7-e2b8fe927906","Type":"ContainerStarted","Data":"f25a7e1d4952678a986c6f1af3779c1ecc5faeb9171ed7890a9697904d913812"}
Jul 09 09:53:46.844941 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.844873    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:53:46.861847 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.861801    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv" podStartSLOduration=1.861786873 podStartE2EDuration="1.861786873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:53:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:53:46.860343928 +0000 UTC m=+1219.185306212" watchObservedRunningTime="2026-07-09 09:53:46.861786873 +0000 UTC m=+1219.186749157"
Jul 09 09:53:46.877509 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:46.877466    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f" podStartSLOduration=0.877451294 podStartE2EDuration="877.451294ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:53:46.875989578 +0000 UTC m=+1219.200951867" watchObservedRunningTime="2026-07-09 09:53:46.877451294 +0000 UTC m=+1219.202413578"
Jul 09 09:53:47.006711 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:47.006676    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:56.006699 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:56.006651    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:56.019374 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:56.019345    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:53:56.894875 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:53:56.894844    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:54:04.491175 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.491146    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"]
Jul 09 09:54:04.494291 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.494274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.496444 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.496408    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 09:54:04.496572 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.496468    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 09 09:54:04.506757 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.506735    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"]
Jul 09 09:54:04.591052 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.591023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tls-certs\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.591242 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.591070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-home\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.591242 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.591092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl4td\" (UniqueName: \"kubernetes.io/projected/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kube-api-access-sl4td\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.591242 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.591138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tmp-dir\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.591242 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.591185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-dshm\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.591242 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.591215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-model-cache\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.591242 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.591234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kserve-provision-location\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.691604 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.691572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tls-certs\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.691805 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.691623    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-home\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.691805 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.691640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl4td\" (UniqueName: \"kubernetes.io/projected/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kube-api-access-sl4td\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.691805 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.691659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tmp-dir\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.691805 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.691678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-dshm\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.691805 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.691708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-model-cache\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.691805 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.691728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kserve-provision-location\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.692147 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.692087    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-home\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.692147 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.692136    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kserve-provision-location\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.692254 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.692182    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tmp-dir\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.692315 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.692263    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-model-cache\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.694012 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.693984    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-dshm\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.694138 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.694123    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tls-certs\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.700042 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.700020    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl4td\" (UniqueName: \"kubernetes.io/projected/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kube-api-access-sl4td\") pod \"auth-disabled-test-kserve-56f8b9c98b-rkwkv\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.804952 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.804865    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:04.960078 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:04.960022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"]
Jul 09 09:54:04.963277 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:54:04.963240    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1bd3d7_12d2_4a0d_8b42_bf8bb8ca0c49.slice/crio-bc25fea1f293ea57ff2543acc1854fbd934c302639ac31ab98a753e4fd84b9bc WatchSource:0}: Error finding container bc25fea1f293ea57ff2543acc1854fbd934c302639ac31ab98a753e4fd84b9bc: Status 404 returned error can't find the container with id bc25fea1f293ea57ff2543acc1854fbd934c302639ac31ab98a753e4fd84b9bc
Jul 09 09:54:05.920435 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:05.920398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" event={"ID":"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49","Type":"ContainerStarted","Data":"502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4"}
Jul 09 09:54:05.921016 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:05.920988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" event={"ID":"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49","Type":"ContainerStarted","Data":"bc25fea1f293ea57ff2543acc1854fbd934c302639ac31ab98a753e4fd84b9bc"}
Jul 09 09:54:08.936014 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:08.935937    2581 generic.go:356] "Generic (PLEG): container finished" podID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerID="502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4" exitCode=0
Jul 09 09:54:08.936364 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:08.936011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" event={"ID":"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49","Type":"ContainerDied","Data":"502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4"}
Jul 09 09:54:09.942833 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:09.942799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" event={"ID":"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49","Type":"ContainerStarted","Data":"56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d"}
Jul 09 09:54:09.967547 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:09.967499    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podStartSLOduration=2.401727813 podStartE2EDuration="5.967482981s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.565755168s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:54:09.965430167 +0000 UTC m=+1242.290392464" watchObservedRunningTime="2026-07-09 09:54:09.967482981 +0000 UTC m=+1242.292445260"
Jul 09 09:54:10.805988 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:10.805952    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:14.805610 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:14.805573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:54:14.807115 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:14.807090    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:54:17.850775 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:17.850745    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:54:24.805507 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:24.805407    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:54:34.806338 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:34.806292    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:54:44.805842 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:44.805800    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:54:48.596602 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.596564    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"]
Jul 09 09:54:48.598753 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.598728    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"]
Jul 09 09:54:48.606130 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:54:48.606079    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 09:54:48.606130 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 09:54:48.606130 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 09:54:48.606130 ip-10-0-129-157 kubenswrapper[2581]:  > containerID="3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4" cmd=["/bin/sleep","15"]
Jul 09 09:54:48.606317 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.606149    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f" message="" err=<
Jul 09 09:54:48.606317 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 09:54:48.606317 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 09:54:48.606317 ip-10-0-129-157 kubenswrapper[2581]:  >
Jul 09 09:54:48.606317 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:54:48.606166    2581 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 09 09:54:48.606317 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 09:54:48.606317 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 09:54:48.606317 ip-10-0-129-157 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f" podUID="90aac38f-36b5-49ce-a0b7-e2b8fe927906" containerName="main" containerID="cri-o://3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4"
Jul 09 09:54:48.606317 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.606220    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f" podUID="90aac38f-36b5-49ce-a0b7-e2b8fe927906" containerName="main" containerID="cri-o://3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4" gracePeriod=60
Jul 09 09:54:48.865023 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.864998    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:54:48.866010 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.865991    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tokenizer-uds\") pod \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") "
Jul 09 09:54:48.866109 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.866030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szcfz\" (UniqueName: \"kubernetes.io/projected/90aac38f-36b5-49ce-a0b7-e2b8fe927906-kube-api-access-szcfz\") pod \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") "
Jul 09 09:54:48.866109 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.866084    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tls-certs\") pod \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\" (UID: \"90aac38f-36b5-49ce-a0b7-e2b8fe927906\") "
Jul 09 09:54:48.866300 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.866275    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "90aac38f-36b5-49ce-a0b7-e2b8fe927906" (UID: "90aac38f-36b5-49ce-a0b7-e2b8fe927906"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:54:48.868156 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.868128    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90aac38f-36b5-49ce-a0b7-e2b8fe927906-kube-api-access-szcfz" (OuterVolumeSpecName: "kube-api-access-szcfz") pod "90aac38f-36b5-49ce-a0b7-e2b8fe927906" (UID: "90aac38f-36b5-49ce-a0b7-e2b8fe927906"). InnerVolumeSpecName "kube-api-access-szcfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:54:48.868276 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.868239    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90aac38f-36b5-49ce-a0b7-e2b8fe927906" (UID: "90aac38f-36b5-49ce-a0b7-e2b8fe927906"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:54:48.966712 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.966665    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tokenizer-uds\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:54:48.966712 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.966692    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szcfz\" (UniqueName: \"kubernetes.io/projected/90aac38f-36b5-49ce-a0b7-e2b8fe927906-kube-api-access-szcfz\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:54:48.966712 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:48.966702    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90aac38f-36b5-49ce-a0b7-e2b8fe927906-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:54:49.099030 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:49.098869    2581 generic.go:356] "Generic (PLEG): container finished" podID="90aac38f-36b5-49ce-a0b7-e2b8fe927906" containerID="3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4" exitCode=0
Jul 09 09:54:49.099030 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:49.098940    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"
Jul 09 09:54:49.099030 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:49.098947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f" event={"ID":"90aac38f-36b5-49ce-a0b7-e2b8fe927906","Type":"ContainerDied","Data":"3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4"}
Jul 09 09:54:49.099030 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:49.099003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f" event={"ID":"90aac38f-36b5-49ce-a0b7-e2b8fe927906","Type":"ContainerDied","Data":"f25a7e1d4952678a986c6f1af3779c1ecc5faeb9171ed7890a9697904d913812"}
Jul 09 09:54:49.099030 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:49.099026    2581 scope.go:117] "RemoveContainer" containerID="3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4"
Jul 09 09:54:49.108442 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:49.108416    2581 scope.go:117] "RemoveContainer" containerID="3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4"
Jul 09 09:54:49.108665 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:54:49.108648    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4\": container with ID starting with 3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4 not found: ID does not exist" containerID="3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4"
Jul 09 09:54:49.108736 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:49.108671    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4"} err="failed to get container status \"3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4\": rpc error: code = NotFound desc = could not find container \"3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4\": container with ID starting with 3d7d441eb9fbe2073e61084f2a1d08b9c0f9c3775b11ab08347b09ef8441e8b4 not found: ID does not exist"
Jul 09 09:54:49.120631 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:49.120602    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"]
Jul 09 09:54:49.124784 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:49.124764    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemz54f"]
Jul 09 09:54:50.228193 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:50.228160    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90aac38f-36b5-49ce-a0b7-e2b8fe927906" path="/var/lib/kubelet/pods/90aac38f-36b5-49ce-a0b7-e2b8fe927906/volumes"
Jul 09 09:54:54.805706 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:54:54.805662    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:55:03.621370 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.621320    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv" podUID="4599ee7b-b7bf-498e-8af5-0e872bb159ad" containerName="main" containerID="cri-o://cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4" gracePeriod=45
Jul 09 09:55:03.868298 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.868268    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:55:03.982919 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.982845    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t2mg\" (UniqueName: \"kubernetes.io/projected/4599ee7b-b7bf-498e-8af5-0e872bb159ad-kube-api-access-5t2mg\") pod \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") "
Jul 09 09:55:03.982919 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.982926    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-dshm\") pod \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") "
Jul 09 09:55:03.983197 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.982950    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tls-certs\") pod \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") "
Jul 09 09:55:03.983197 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.982973    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-model-cache\") pod \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") "
Jul 09 09:55:03.983197 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.982996    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-home\") pod \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") "
Jul 09 09:55:03.983197 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.983042    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tmp-dir\") pod \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\" (UID: \"4599ee7b-b7bf-498e-8af5-0e872bb159ad\") "
Jul 09 09:55:03.983410 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.983239    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-home" (OuterVolumeSpecName: "home") pod "4599ee7b-b7bf-498e-8af5-0e872bb159ad" (UID: "4599ee7b-b7bf-498e-8af5-0e872bb159ad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:55:03.983410 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.983270    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-model-cache" (OuterVolumeSpecName: "model-cache") pod "4599ee7b-b7bf-498e-8af5-0e872bb159ad" (UID: "4599ee7b-b7bf-498e-8af5-0e872bb159ad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:55:03.983410 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.983306    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4599ee7b-b7bf-498e-8af5-0e872bb159ad" (UID: "4599ee7b-b7bf-498e-8af5-0e872bb159ad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:55:03.983572 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.983461    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:55:03.983572 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.983475    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:55:03.983572 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.983484    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:55:03.985101 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.985066    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-dshm" (OuterVolumeSpecName: "dshm") pod "4599ee7b-b7bf-498e-8af5-0e872bb159ad" (UID: "4599ee7b-b7bf-498e-8af5-0e872bb159ad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:55:03.985444 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.985422    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4599ee7b-b7bf-498e-8af5-0e872bb159ad" (UID: "4599ee7b-b7bf-498e-8af5-0e872bb159ad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:55:03.985512 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:03.985448    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4599ee7b-b7bf-498e-8af5-0e872bb159ad-kube-api-access-5t2mg" (OuterVolumeSpecName: "kube-api-access-5t2mg") pod "4599ee7b-b7bf-498e-8af5-0e872bb159ad" (UID: "4599ee7b-b7bf-498e-8af5-0e872bb159ad"). InnerVolumeSpecName "kube-api-access-5t2mg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:55:04.084192 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.084162    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4599ee7b-b7bf-498e-8af5-0e872bb159ad-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:55:04.084192 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.084188    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4599ee7b-b7bf-498e-8af5-0e872bb159ad-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:55:04.084192 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.084201    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t2mg\" (UniqueName: \"kubernetes.io/projected/4599ee7b-b7bf-498e-8af5-0e872bb159ad-kube-api-access-5t2mg\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:55:04.163232 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.163200    2581 generic.go:356] "Generic (PLEG): container finished" podID="4599ee7b-b7bf-498e-8af5-0e872bb159ad" containerID="cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4" exitCode=0
Jul 09 09:55:04.163411 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.163261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv" event={"ID":"4599ee7b-b7bf-498e-8af5-0e872bb159ad","Type":"ContainerDied","Data":"cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4"}
Jul 09 09:55:04.163411 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.163262    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"
Jul 09 09:55:04.163411 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.163303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv" event={"ID":"4599ee7b-b7bf-498e-8af5-0e872bb159ad","Type":"ContainerDied","Data":"fa926baee1a184af60b52ef7ed5da3dff7a8fc2f9e2002b5cd2b6d21406bb73b"}
Jul 09 09:55:04.163411 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.163325    2581 scope.go:117] "RemoveContainer" containerID="cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4"
Jul 09 09:55:04.171588 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.171570    2581 scope.go:117] "RemoveContainer" containerID="cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4"
Jul 09 09:55:04.171842 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:55:04.171818    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4\": container with ID starting with cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4 not found: ID does not exist" containerID="cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4"
Jul 09 09:55:04.171908 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.171851    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4"} err="failed to get container status \"cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4\": rpc error: code = NotFound desc = could not find container \"cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4\": container with ID starting with cedc046b76b7d3714b007c540f8f4614d300ec32e0e2fd68320c514d3779c9b4 not found: ID does not exist"
Jul 09 09:55:04.186752 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.186718    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"]
Jul 09 09:55:04.193165 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.193140    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-5c54ddb98fnpcxv"]
Jul 09 09:55:04.227715 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.227678    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4599ee7b-b7bf-498e-8af5-0e872bb159ad" path="/var/lib/kubelet/pods/4599ee7b-b7bf-498e-8af5-0e872bb159ad/volumes"
Jul 09 09:55:04.805477 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:04.805421    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:55:14.805872 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:14.805822    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:55:24.805570 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:24.805532    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:55:34.805841 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:34.805803    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:55:42.877479 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.877442    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"]
Jul 09 09:55:42.878066 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.877826    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4599ee7b-b7bf-498e-8af5-0e872bb159ad" containerName="main"
Jul 09 09:55:42.878066 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.877841    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4599ee7b-b7bf-498e-8af5-0e872bb159ad" containerName="main"
Jul 09 09:55:42.878066 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.877857    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90aac38f-36b5-49ce-a0b7-e2b8fe927906" containerName="main"
Jul 09 09:55:42.878066 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.877865    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="90aac38f-36b5-49ce-a0b7-e2b8fe927906" containerName="main"
Jul 09 09:55:42.878066 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.877965    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4599ee7b-b7bf-498e-8af5-0e872bb159ad" containerName="main"
Jul 09 09:55:42.878066 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.877978    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="90aac38f-36b5-49ce-a0b7-e2b8fe927906" containerName="main"
Jul 09 09:55:42.881214 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.881194    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:42.883846 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.883822    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 09 09:55:42.883979 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.883863    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 09:55:42.895301 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.895275    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"]
Jul 09 09:55:42.999324 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.999283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19fc8c88-47f4-43e7-a0ef-072dea8673db-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:42.999511 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.999355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:42.999511 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.999395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88l69\" (UniqueName: \"kubernetes.io/projected/19fc8c88-47f4-43e7-a0ef-072dea8673db-kube-api-access-88l69\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:42.999511 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.999421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:42.999511 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.999442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:42.999511 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.999462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:42.999511 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.999494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19fc8c88-47f4-43e7-a0ef-072dea8673db-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:42.999719 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:42.999550    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.100685 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.100635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.100878 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.100702    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88l69\" (UniqueName: \"kubernetes.io/projected/19fc8c88-47f4-43e7-a0ef-072dea8673db-kube-api-access-88l69\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.100878 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.100734    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.100878 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.100760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.100878 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.100787    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.100878 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.100834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19fc8c88-47f4-43e7-a0ef-072dea8673db-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.100878 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.100862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.101211 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.100917    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19fc8c88-47f4-43e7-a0ef-072dea8673db-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.101211 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.101147    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.101211 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.101193    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.101332 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.101309    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.101368 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.101351    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.101669 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.101644    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19fc8c88-47f4-43e7-a0ef-072dea8673db-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.103301 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.103269    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.103565 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.103546    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19fc8c88-47f4-43e7-a0ef-072dea8673db-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.108997 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.108975    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88l69\" (UniqueName: \"kubernetes.io/projected/19fc8c88-47f4-43e7-a0ef-072dea8673db-kube-api-access-88l69\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.191276 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.191172    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:43.330090 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:43.330055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"]
Jul 09 09:55:43.332704 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:55:43.332678    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fc8c88_47f4_43e7_a0ef_072dea8673db.slice/crio-b68fdbef4da69bd1fd79c4867c79a58fb36ff2868242ef5eebd550a8608f5c79 WatchSource:0}: Error finding container b68fdbef4da69bd1fd79c4867c79a58fb36ff2868242ef5eebd550a8608f5c79: Status 404 returned error can't find the container with id b68fdbef4da69bd1fd79c4867c79a58fb36ff2868242ef5eebd550a8608f5c79
Jul 09 09:55:44.327109 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:44.327063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" event={"ID":"19fc8c88-47f4-43e7-a0ef-072dea8673db","Type":"ContainerStarted","Data":"cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa"}
Jul 09 09:55:44.327659 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:44.327131    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" event={"ID":"19fc8c88-47f4-43e7-a0ef-072dea8673db","Type":"ContainerStarted","Data":"b68fdbef4da69bd1fd79c4867c79a58fb36ff2868242ef5eebd550a8608f5c79"}
Jul 09 09:55:44.805463 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:44.805424    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:55:47.342752 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:47.342717    2581 generic.go:356] "Generic (PLEG): container finished" podID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerID="cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa" exitCode=0
Jul 09 09:55:47.343111 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:47.342765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" event={"ID":"19fc8c88-47f4-43e7-a0ef-072dea8673db","Type":"ContainerDied","Data":"cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa"}
Jul 09 09:55:48.350285 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:48.350237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" event={"ID":"19fc8c88-47f4-43e7-a0ef-072dea8673db","Type":"ContainerStarted","Data":"8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7"}
Jul 09 09:55:48.376406 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:48.376335    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" podStartSLOduration=2.477269806 podStartE2EDuration="6.376312435s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.899042629s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:55:48.374646727 +0000 UTC m=+1340.699609010" watchObservedRunningTime="2026-07-09 09:55:48.376312435 +0000 UTC m=+1340.701274801"
Jul 09 09:55:49.192061 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:49.192010    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:53.192413 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:53.192315    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:55:53.194164 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:53.194134    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 09:55:54.805706 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:55:54.805657    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": dial tcp 10.133.0.33:8000: connect: connection refused"
Jul 09 09:56:03.191993 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:03.191953    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 09:56:04.815271 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:04.815237    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:56:04.823043 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:04.823014    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:56:10.003932 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:10.003877    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"]
Jul 09 09:56:13.192729 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:13.192684    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 09:56:23.192290 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:23.192245    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 09:56:25.027850 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.027805    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main" containerID="cri-o://56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d" gracePeriod=45
Jul 09 09:56:25.775359 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.775336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:56:25.851677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.851591    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-home\") pod \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") "
Jul 09 09:56:25.851677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.851646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tmp-dir\") pod \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") "
Jul 09 09:56:25.851915 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.851680    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sl4td\" (UniqueName: \"kubernetes.io/projected/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kube-api-access-sl4td\") pod \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") "
Jul 09 09:56:25.851915 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.851722    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tls-certs\") pod \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") "
Jul 09 09:56:25.851915 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.851743    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-dshm\") pod \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") "
Jul 09 09:56:25.851915 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.851790    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-model-cache\") pod \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") "
Jul 09 09:56:25.851915 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.851812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kserve-provision-location\") pod \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\" (UID: \"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49\") "
Jul 09 09:56:25.852442 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.852384    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" (UID: "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:56:25.852545 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.852470    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-home" (OuterVolumeSpecName: "home") pod "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" (UID: "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:56:25.854318 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.854289    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-dshm" (OuterVolumeSpecName: "dshm") pod "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" (UID: "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:56:25.854430 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.854339    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kube-api-access-sl4td" (OuterVolumeSpecName: "kube-api-access-sl4td") pod "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" (UID: "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49"). InnerVolumeSpecName "kube-api-access-sl4td". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:56:25.854430 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.854344    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" (UID: "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:56:25.864359 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.864335    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" (UID: "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:56:25.912819 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.912773    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" (UID: "0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:56:25.952737 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.952703    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:56:25.952737 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.952732    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:56:25.952737 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.952743    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl4td\" (UniqueName: \"kubernetes.io/projected/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kube-api-access-sl4td\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:56:25.952973 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.952751    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:56:25.952973 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.952760    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:56:25.952973 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.952767    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:56:25.952973 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:25.952776    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:56:26.505484 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.505384    2581 generic.go:356] "Generic (PLEG): container finished" podID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerID="56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d" exitCode=0
Jul 09 09:56:26.505484 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.505444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" event={"ID":"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49","Type":"ContainerDied","Data":"56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d"}
Jul 09 09:56:26.505484 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.505448    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"
Jul 09 09:56:26.506045 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.505488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv" event={"ID":"0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49","Type":"ContainerDied","Data":"bc25fea1f293ea57ff2543acc1854fbd934c302639ac31ab98a753e4fd84b9bc"}
Jul 09 09:56:26.506045 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.505512    2581 scope.go:117] "RemoveContainer" containerID="56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d"
Jul 09 09:56:26.513802 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.513769    2581 scope.go:117] "RemoveContainer" containerID="502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4"
Jul 09 09:56:26.522259 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.522230    2581 scope.go:117] "RemoveContainer" containerID="56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d"
Jul 09 09:56:26.522506 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:56:26.522487    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d\": container with ID starting with 56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d not found: ID does not exist" containerID="56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d"
Jul 09 09:56:26.522590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.522513    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d"} err="failed to get container status \"56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d\": rpc error: code = NotFound desc = could not find container \"56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d\": container with ID starting with 56ffa6ee26bb08a9f0c1df86aa0ba27f85c8d6f04aa4ef6742ce3eada695c82d not found: ID does not exist"
Jul 09 09:56:26.522590 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.522529    2581 scope.go:117] "RemoveContainer" containerID="502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4"
Jul 09 09:56:26.522788 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:56:26.522772    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4\": container with ID starting with 502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4 not found: ID does not exist" containerID="502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4"
Jul 09 09:56:26.522830 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.522794    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4"} err="failed to get container status \"502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4\": rpc error: code = NotFound desc = could not find container \"502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4\": container with ID starting with 502f675343818d078f42423603fda99b568bc9e906ad7c8a88e37be1cefef2a4 not found: ID does not exist"
Jul 09 09:56:26.525605 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.525586    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"]
Jul 09 09:56:26.531545 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:26.531526    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-56f8b9c98b-rkwkv"]
Jul 09 09:56:28.227714 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:28.227681    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" path="/var/lib/kubelet/pods/0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49/volumes"
Jul 09 09:56:33.192510 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:33.192474    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 09:56:43.192446 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:43.192403    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 09:56:53.193028 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:56:53.192979    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 09:57:03.202635 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:03.202605    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:57:03.210504 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:03.210468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:57:27.820709 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:27.820623    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"]
Jul 09 09:57:42.843243 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:42.843207    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main" containerID="cri-o://8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7" gracePeriod=45
Jul 09 09:57:43.204183 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.204145    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/health\": dial tcp 10.133.0.34:8000: connect: connection refused"
Jul 09 09:57:43.494135 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.494110    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:57:43.603807 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.603779    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19fc8c88-47f4-43e7-a0ef-072dea8673db-cabundle-cert\") pod \"19fc8c88-47f4-43e7-a0ef-072dea8673db\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") "
Jul 09 09:57:43.603807 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.603812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88l69\" (UniqueName: \"kubernetes.io/projected/19fc8c88-47f4-43e7-a0ef-072dea8673db-kube-api-access-88l69\") pod \"19fc8c88-47f4-43e7-a0ef-072dea8673db\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") "
Jul 09 09:57:43.604040 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.603842    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-model-cache\") pod \"19fc8c88-47f4-43e7-a0ef-072dea8673db\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") "
Jul 09 09:57:43.604040 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.603966    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-home\") pod \"19fc8c88-47f4-43e7-a0ef-072dea8673db\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") "
Jul 09 09:57:43.604134 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.604060    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-kserve-provision-location\") pod \"19fc8c88-47f4-43e7-a0ef-072dea8673db\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") "
Jul 09 09:57:43.604190 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.604149    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-tmp-dir\") pod \"19fc8c88-47f4-43e7-a0ef-072dea8673db\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") "
Jul 09 09:57:43.604190 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.604155    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-model-cache" (OuterVolumeSpecName: "model-cache") pod "19fc8c88-47f4-43e7-a0ef-072dea8673db" (UID: "19fc8c88-47f4-43e7-a0ef-072dea8673db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:57:43.604190 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.604178    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-dshm\") pod \"19fc8c88-47f4-43e7-a0ef-072dea8673db\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") "
Jul 09 09:57:43.604351 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.604219    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19fc8c88-47f4-43e7-a0ef-072dea8673db-tls-certs\") pod \"19fc8c88-47f4-43e7-a0ef-072dea8673db\" (UID: \"19fc8c88-47f4-43e7-a0ef-072dea8673db\") "
Jul 09 09:57:43.604454 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.604435    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:57:43.604517 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.604429    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-home" (OuterVolumeSpecName: "home") pod "19fc8c88-47f4-43e7-a0ef-072dea8673db" (UID: "19fc8c88-47f4-43e7-a0ef-072dea8673db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:57:43.604624 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.604585    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fc8c88-47f4-43e7-a0ef-072dea8673db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19fc8c88-47f4-43e7-a0ef-072dea8673db" (UID: "19fc8c88-47f4-43e7-a0ef-072dea8673db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 09:57:43.606083 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.606053    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fc8c88-47f4-43e7-a0ef-072dea8673db-kube-api-access-88l69" (OuterVolumeSpecName: "kube-api-access-88l69") pod "19fc8c88-47f4-43e7-a0ef-072dea8673db" (UID: "19fc8c88-47f4-43e7-a0ef-072dea8673db"). InnerVolumeSpecName "kube-api-access-88l69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 09:57:43.606295 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.606264    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-dshm" (OuterVolumeSpecName: "dshm") pod "19fc8c88-47f4-43e7-a0ef-072dea8673db" (UID: "19fc8c88-47f4-43e7-a0ef-072dea8673db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:57:43.606774 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.606755    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fc8c88-47f4-43e7-a0ef-072dea8673db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19fc8c88-47f4-43e7-a0ef-072dea8673db" (UID: "19fc8c88-47f4-43e7-a0ef-072dea8673db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 09:57:43.616272 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.616248    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19fc8c88-47f4-43e7-a0ef-072dea8673db" (UID: "19fc8c88-47f4-43e7-a0ef-072dea8673db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:57:43.665745 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.665706    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19fc8c88-47f4-43e7-a0ef-072dea8673db" (UID: "19fc8c88-47f4-43e7-a0ef-072dea8673db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 09:57:43.705657 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.705625    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:57:43.705657 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.705655    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:57:43.705761 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.705663    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19fc8c88-47f4-43e7-a0ef-072dea8673db-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:57:43.705761 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.705671    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19fc8c88-47f4-43e7-a0ef-072dea8673db-cabundle-cert\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:57:43.705761 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.705680    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88l69\" (UniqueName: \"kubernetes.io/projected/19fc8c88-47f4-43e7-a0ef-072dea8673db-kube-api-access-88l69\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:57:43.705761 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.705688    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:57:43.705761 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.705696    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19fc8c88-47f4-43e7-a0ef-072dea8673db-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 09:57:43.813130 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.813053    2581 generic.go:356] "Generic (PLEG): container finished" podID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerID="8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7" exitCode=0
Jul 09 09:57:43.813130 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.813109    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"
Jul 09 09:57:43.813287 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.813135    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" event={"ID":"19fc8c88-47f4-43e7-a0ef-072dea8673db","Type":"ContainerDied","Data":"8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7"}
Jul 09 09:57:43.813287 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.813188    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms" event={"ID":"19fc8c88-47f4-43e7-a0ef-072dea8673db","Type":"ContainerDied","Data":"b68fdbef4da69bd1fd79c4867c79a58fb36ff2868242ef5eebd550a8608f5c79"}
Jul 09 09:57:43.813287 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.813209    2581 scope.go:117] "RemoveContainer" containerID="8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7"
Jul 09 09:57:43.821962 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.821930    2581 scope.go:117] "RemoveContainer" containerID="cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa"
Jul 09 09:57:43.829970 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.829953    2581 scope.go:117] "RemoveContainer" containerID="8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7"
Jul 09 09:57:43.830241 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:57:43.830223    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7\": container with ID starting with 8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7 not found: ID does not exist" containerID="8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7"
Jul 09 09:57:43.830281 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.830252    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7"} err="failed to get container status \"8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7\": rpc error: code = NotFound desc = could not find container \"8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7\": container with ID starting with 8491ef33e7897dafb07d790e3fc9767dc8397834f5e5bd72a30196635e50d0a7 not found: ID does not exist"
Jul 09 09:57:43.830281 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.830270    2581 scope.go:117] "RemoveContainer" containerID="cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa"
Jul 09 09:57:43.830502 ip-10-0-129-157 kubenswrapper[2581]: E0709 09:57:43.830484    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa\": container with ID starting with cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa not found: ID does not exist" containerID="cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa"
Jul 09 09:57:43.830538 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.830507    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa"} err="failed to get container status \"cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa\": rpc error: code = NotFound desc = could not find container \"cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa\": container with ID starting with cad13c15c2cd91d1cf08bcdcdc66e373c300f35a3e238e553a7c9ba886bae9aa not found: ID does not exist"
Jul 09 09:57:43.836239 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.836213    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"]
Jul 09 09:57:43.842832 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:43.839548    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c9f69cd9c-b5nms"]
Jul 09 09:57:44.227636 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:44.227607    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" path="/var/lib/kubelet/pods/19fc8c88-47f4-43e7-a0ef-072dea8673db/volumes"
Jul 09 09:57:53.003947 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.003914    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"]
Jul 09 09:57:53.004311 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.004180    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="storage-initializer"
Jul 09 09:57:53.004311 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.004192    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="storage-initializer"
Jul 09 09:57:53.004311 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.004202    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main"
Jul 09 09:57:53.004311 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.004207    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main"
Jul 09 09:57:53.004311 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.004213    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main"
Jul 09 09:57:53.004311 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.004219    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main"
Jul 09 09:57:53.004311 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.004231    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="storage-initializer"
Jul 09 09:57:53.004311 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.004236    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="storage-initializer"
Jul 09 09:57:53.004311 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.004283    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f1bd3d7-12d2-4a0d-8b42-bf8bb8ca0c49" containerName="main"
Jul 09 09:57:53.004311 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.004290    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="19fc8c88-47f4-43e7-a0ef-072dea8673db" containerName="main"
Jul 09 09:57:53.009865 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.009848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.012764 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.012740    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 09:57:53.012924 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.012839    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 09:57:53.013164 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.013148    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 09 09:57:53.013492 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.013476    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 09:57:53.013492 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.013484    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkhz8\""
Jul 09 09:57:53.013619 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.013484    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 09:57:53.024420 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.024399    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"]
Jul 09 09:57:53.069227 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.069201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.069389 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.069235    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.069389 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.069259    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdqr2\" (UniqueName: \"kubernetes.io/projected/55ed1aca-6f43-4140-ba64-76457b1c54ff-kube-api-access-sdqr2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.069389 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.069287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.069389 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.069303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55ed1aca-6f43-4140-ba64-76457b1c54ff-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.069389 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.069330    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.069389 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.069347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ed1aca-6f43-4140-ba64-76457b1c54ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.069606 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.069414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.170397 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.170361    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.170397 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.170402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdqr2\" (UniqueName: \"kubernetes.io/projected/55ed1aca-6f43-4140-ba64-76457b1c54ff-kube-api-access-sdqr2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.170658 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.170422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.170658 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.170439    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55ed1aca-6f43-4140-ba64-76457b1c54ff-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.170658 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.170476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.170658 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.170496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ed1aca-6f43-4140-ba64-76457b1c54ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.170658 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.170517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.170933 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.170773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.170933 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.170840    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.170933 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.170854    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.171098 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.171033    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.171161 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.171127    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.171273 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.171254    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55ed1aca-6f43-4140-ba64-76457b1c54ff-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.172831 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.172808    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.172988 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.172970    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ed1aca-6f43-4140-ba64-76457b1c54ff-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.178754 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.178732    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdqr2\" (UniqueName: \"kubernetes.io/projected/55ed1aca-6f43-4140-ba64-76457b1c54ff-kube-api-access-sdqr2\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.319748 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.319670    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:57:53.451664 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.451638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"]
Jul 09 09:57:53.453676 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:57:53.453649    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ed1aca_6f43_4140_ba64_76457b1c54ff.slice/crio-adc7c94b2e3f2503db11bbb8c1030f508c9815e6c5bf2eb38e1e95c9649e5dc0 WatchSource:0}: Error finding container adc7c94b2e3f2503db11bbb8c1030f508c9815e6c5bf2eb38e1e95c9649e5dc0: Status 404 returned error can't find the container with id adc7c94b2e3f2503db11bbb8c1030f508c9815e6c5bf2eb38e1e95c9649e5dc0
Jul 09 09:57:53.847554 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.847510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" event={"ID":"55ed1aca-6f43-4140-ba64-76457b1c54ff","Type":"ContainerStarted","Data":"d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5"}
Jul 09 09:57:53.847729 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:53.847577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" event={"ID":"55ed1aca-6f43-4140-ba64-76457b1c54ff","Type":"ContainerStarted","Data":"adc7c94b2e3f2503db11bbb8c1030f508c9815e6c5bf2eb38e1e95c9649e5dc0"}
Jul 09 09:57:57.864345 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:57.864267    2581 generic.go:356] "Generic (PLEG): container finished" podID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerID="d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5" exitCode=0
Jul 09 09:57:57.864720 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:57.864343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" event={"ID":"55ed1aca-6f43-4140-ba64-76457b1c54ff","Type":"ContainerDied","Data":"d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5"}
Jul 09 09:57:58.870277 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:58.870241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" event={"ID":"55ed1aca-6f43-4140-ba64-76457b1c54ff","Type":"ContainerStarted","Data":"35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7"}
Jul 09 09:57:58.903078 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:58.903031    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" podStartSLOduration=2.8799559439999998 podStartE2EDuration="6.903016627s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.023060683s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:57:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:57:58.900498705 +0000 UTC m=+1471.225460990" watchObservedRunningTime="2026-07-09 09:57:58.903016627 +0000 UTC m=+1471.227978910"
Jul 09 09:57:59.319804 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:57:59.319771    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:58:03.320696 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:03.320649    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:58:03.322344 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:03.322310    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 09:58:13.320328 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:13.320281    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 09:58:23.320469 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:23.320411    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 09:58:30.020580 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:30.020547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:58:30.022140 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:30.022123    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 09:58:33.320088 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:33.320051    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 09:58:40.907171 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:40.907135    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"]
Jul 09 09:58:40.910662 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:40.910640    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:40.913204 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:40.913185    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 09 09:58:40.921928 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:40.921901    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"]
Jul 09 09:58:41.057323 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.057283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.057519 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.057347    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.057519 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.057391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.057519 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.057428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.057519 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.057460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.057519 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.057487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml55h\" (UniqueName: \"kubernetes.io/projected/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kube-api-access-ml55h\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.057701 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.057522    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.158341 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.158256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.158485 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.158387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.158485 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.158430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.158485 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.158459    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.158612 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.158493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml55h\" (UniqueName: \"kubernetes.io/projected/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kube-api-access-ml55h\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.158612 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.158533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.158612 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.158603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.159006 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.158790    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.159006 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.158918    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.159006 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.158999    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.159202 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.159068    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.160677 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.160650    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.161025 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.161008    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.167165 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.167139    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml55h\" (UniqueName: \"kubernetes.io/projected/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kube-api-access-ml55h\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.222560 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.222521    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:41.360182 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:41.360145    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"]
Jul 09 09:58:41.363072 ip-10-0-129-157 kubenswrapper[2581]: W0709 09:58:41.363036    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c37ff3_c6de_4852_97c8_58c28cbd3944.slice/crio-55cee949878cd9794babdeb2914fc766118bb9bfebf49c9b8ad5afc14d190aa1 WatchSource:0}: Error finding container 55cee949878cd9794babdeb2914fc766118bb9bfebf49c9b8ad5afc14d190aa1: Status 404 returned error can't find the container with id 55cee949878cd9794babdeb2914fc766118bb9bfebf49c9b8ad5afc14d190aa1
Jul 09 09:58:42.036990 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:42.036931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" event={"ID":"a6c37ff3-c6de-4852-97c8-58c28cbd3944","Type":"ContainerStarted","Data":"754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db"}
Jul 09 09:58:42.037460 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:42.037017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" event={"ID":"a6c37ff3-c6de-4852-97c8-58c28cbd3944","Type":"ContainerStarted","Data":"55cee949878cd9794babdeb2914fc766118bb9bfebf49c9b8ad5afc14d190aa1"}
Jul 09 09:58:43.321044 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:43.320940    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 09:58:45.052871 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:45.052838    2581 generic.go:356] "Generic (PLEG): container finished" podID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerID="754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db" exitCode=0
Jul 09 09:58:45.053281 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:45.052913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" event={"ID":"a6c37ff3-c6de-4852-97c8-58c28cbd3944","Type":"ContainerDied","Data":"754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db"}
Jul 09 09:58:46.058672 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:46.058635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" event={"ID":"a6c37ff3-c6de-4852-97c8-58c28cbd3944","Type":"ContainerStarted","Data":"f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61"}
Jul 09 09:58:46.085410 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:46.085360    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podStartSLOduration=2.889814666 podStartE2EDuration="6.085339326s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.19552466s" isStatefulPod=false podCreationTimestamp="2026-07-09 09:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 09:58:46.085167058 +0000 UTC m=+1518.410129344" watchObservedRunningTime="2026-07-09 09:58:46.085339326 +0000 UTC m=+1518.410301611"
Jul 09 09:58:46.228717 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:46.228681    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:51.223404 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:51.223301    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 09:58:51.224999 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:51.224971    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 09:58:53.320460 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:58:53.320417    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 09:59:01.223089 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:59:01.223047    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 09:59:03.321104 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:59:03.321047    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 09:59:11.223212 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:59:11.223158    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 09:59:13.330106 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:59:13.330074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:59:13.337846 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:59:13.337823    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 09:59:21.223555 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:59:21.223506    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 09:59:31.223721 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:59:31.223675    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 09:59:41.223298 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:59:41.223255    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 09:59:51.223001 ip-10-0-129-157 kubenswrapper[2581]: I0709 09:59:51.222957    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 10:00:01.223173 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:01.223124    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 10:00:11.223866 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:11.223825    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 10:00:14.668117 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:14.668081    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"]
Jul 09 10:00:21.223289 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:21.223198    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 10:00:29.690247 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:29.690203    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main" containerID="cri-o://35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7" gracePeriod=45
Jul 09 10:00:30.331348 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.331306    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/health\": dial tcp 10.133.0.35:8000: connect: connection refused"
Jul 09 10:00:30.445487 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.445449    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 10:00:30.489747 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.489709    2581 generic.go:356] "Generic (PLEG): container finished" podID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerID="35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7" exitCode=0
Jul 09 10:00:30.489977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.489762    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"
Jul 09 10:00:30.489977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.489772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" event={"ID":"55ed1aca-6f43-4140-ba64-76457b1c54ff","Type":"ContainerDied","Data":"35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7"}
Jul 09 10:00:30.489977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.489866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9" event={"ID":"55ed1aca-6f43-4140-ba64-76457b1c54ff","Type":"ContainerDied","Data":"adc7c94b2e3f2503db11bbb8c1030f508c9815e6c5bf2eb38e1e95c9649e5dc0"}
Jul 09 10:00:30.489977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.489918    2581 scope.go:117] "RemoveContainer" containerID="35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7"
Jul 09 10:00:30.499539 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.499519    2581 scope.go:117] "RemoveContainer" containerID="d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5"
Jul 09 10:00:30.507375 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.507351    2581 scope.go:117] "RemoveContainer" containerID="35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7"
Jul 09 10:00:30.507658 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:00:30.507633    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7\": container with ID starting with 35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7 not found: ID does not exist" containerID="35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7"
Jul 09 10:00:30.507771 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.507666    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7"} err="failed to get container status \"35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7\": rpc error: code = NotFound desc = could not find container \"35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7\": container with ID starting with 35b165b076d81e395c63e0e8de4c21429572c704fb1022726c24f250f671c9d7 not found: ID does not exist"
Jul 09 10:00:30.507771 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.507684    2581 scope.go:117] "RemoveContainer" containerID="d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5"
Jul 09 10:00:30.507961 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:00:30.507935    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5\": container with ID starting with d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5 not found: ID does not exist" containerID="d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5"
Jul 09 10:00:30.508047 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.507969    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5"} err="failed to get container status \"d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5\": rpc error: code = NotFound desc = could not find container \"d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5\": container with ID starting with d1349351db0ab55475a14cb55a022122b5c1405181f6e129719ddb74da8a58d5 not found: ID does not exist"
Jul 09 10:00:30.509228 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509208    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ed1aca-6f43-4140-ba64-76457b1c54ff-tls-certs\") pod \"55ed1aca-6f43-4140-ba64-76457b1c54ff\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") "
Jul 09 10:00:30.509328 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509259    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-home\") pod \"55ed1aca-6f43-4140-ba64-76457b1c54ff\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") "
Jul 09 10:00:30.509328 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509284    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-model-cache\") pod \"55ed1aca-6f43-4140-ba64-76457b1c54ff\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") "
Jul 09 10:00:30.509328 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509322    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-tmp-dir\") pod \"55ed1aca-6f43-4140-ba64-76457b1c54ff\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") "
Jul 09 10:00:30.509477 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdqr2\" (UniqueName: \"kubernetes.io/projected/55ed1aca-6f43-4140-ba64-76457b1c54ff-kube-api-access-sdqr2\") pod \"55ed1aca-6f43-4140-ba64-76457b1c54ff\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") "
Jul 09 10:00:30.509477 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509418    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-dshm\") pod \"55ed1aca-6f43-4140-ba64-76457b1c54ff\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") "
Jul 09 10:00:30.509572 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-kserve-provision-location\") pod \"55ed1aca-6f43-4140-ba64-76457b1c54ff\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") "
Jul 09 10:00:30.509627 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509561    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "55ed1aca-6f43-4140-ba64-76457b1c54ff" (UID: "55ed1aca-6f43-4140-ba64-76457b1c54ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:00:30.509627 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509580    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55ed1aca-6f43-4140-ba64-76457b1c54ff-cabundle-cert\") pod \"55ed1aca-6f43-4140-ba64-76457b1c54ff\" (UID: \"55ed1aca-6f43-4140-ba64-76457b1c54ff\") "
Jul 09 10:00:30.509806 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509770    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-home" (OuterVolumeSpecName: "home") pod "55ed1aca-6f43-4140-ba64-76457b1c54ff" (UID: "55ed1aca-6f43-4140-ba64-76457b1c54ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:00:30.509924 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509799    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55ed1aca-6f43-4140-ba64-76457b1c54ff" (UID: "55ed1aca-6f43-4140-ba64-76457b1c54ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:00:30.509924 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509843    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:00:30.509924 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.509863    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:00:30.510128 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.510106    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ed1aca-6f43-4140-ba64-76457b1c54ff-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "55ed1aca-6f43-4140-ba64-76457b1c54ff" (UID: "55ed1aca-6f43-4140-ba64-76457b1c54ff"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:00:30.511449 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.511425    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ed1aca-6f43-4140-ba64-76457b1c54ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55ed1aca-6f43-4140-ba64-76457b1c54ff" (UID: "55ed1aca-6f43-4140-ba64-76457b1c54ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:00:30.511544 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.511527    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ed1aca-6f43-4140-ba64-76457b1c54ff-kube-api-access-sdqr2" (OuterVolumeSpecName: "kube-api-access-sdqr2") pod "55ed1aca-6f43-4140-ba64-76457b1c54ff" (UID: "55ed1aca-6f43-4140-ba64-76457b1c54ff"). InnerVolumeSpecName "kube-api-access-sdqr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:00:30.512179 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.512157    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-dshm" (OuterVolumeSpecName: "dshm") pod "55ed1aca-6f43-4140-ba64-76457b1c54ff" (UID: "55ed1aca-6f43-4140-ba64-76457b1c54ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:00:30.576809 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.576760    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "55ed1aca-6f43-4140-ba64-76457b1c54ff" (UID: "55ed1aca-6f43-4140-ba64-76457b1c54ff"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:00:30.610399 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.610363    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:00:30.610399 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.610389    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/55ed1aca-6f43-4140-ba64-76457b1c54ff-cabundle-cert\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:00:30.610399 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.610400    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55ed1aca-6f43-4140-ba64-76457b1c54ff-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:00:30.610399 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.610407    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:00:30.610784 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.610415    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdqr2\" (UniqueName: \"kubernetes.io/projected/55ed1aca-6f43-4140-ba64-76457b1c54ff-kube-api-access-sdqr2\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:00:30.610784 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.610423    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55ed1aca-6f43-4140-ba64-76457b1c54ff-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:00:30.813473 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.813430    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"]
Jul 09 10:00:30.818362 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:30.818333    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-f865f855d-mmkz9"]
Jul 09 10:00:31.223188 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:31.223146    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 10:00:32.227900 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:32.227841    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" path="/var/lib/kubelet/pods/55ed1aca-6f43-4140-ba64-76457b1c54ff/volumes"
Jul 09 10:00:41.232471 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:41.232440    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 10:00:41.240626 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:41.240604    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 10:00:45.660833 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:00:45.660801    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"]
Jul 09 10:01:00.683722 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:00.683681    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" containerID="cri-o://f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61" gracePeriod=45
Jul 09 10:01:01.233499 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.233461    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/health\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 10:01:01.337717 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.337690    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 10:01:01.464483 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.464455    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-dshm\") pod \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") "
Jul 09 10:01:01.464658 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.464514    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kserve-provision-location\") pod \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") "
Jul 09 10:01:01.464658 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.464548    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tls-certs\") pod \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") "
Jul 09 10:01:01.464658 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.464567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ml55h\" (UniqueName: \"kubernetes.io/projected/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kube-api-access-ml55h\") pod \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") "
Jul 09 10:01:01.464658 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.464586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-model-cache\") pod \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") "
Jul 09 10:01:01.464658 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.464605    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-home\") pod \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") "
Jul 09 10:01:01.464658 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.464651    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tmp-dir\") pod \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\" (UID: \"a6c37ff3-c6de-4852-97c8-58c28cbd3944\") "
Jul 09 10:01:01.465094 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.464942    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6c37ff3-c6de-4852-97c8-58c28cbd3944" (UID: "a6c37ff3-c6de-4852-97c8-58c28cbd3944"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:01:01.465383 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.465346    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-home" (OuterVolumeSpecName: "home") pod "a6c37ff3-c6de-4852-97c8-58c28cbd3944" (UID: "a6c37ff3-c6de-4852-97c8-58c28cbd3944"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:01:01.466704 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.466676    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-dshm" (OuterVolumeSpecName: "dshm") pod "a6c37ff3-c6de-4852-97c8-58c28cbd3944" (UID: "a6c37ff3-c6de-4852-97c8-58c28cbd3944"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:01:01.466810 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.466722    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kube-api-access-ml55h" (OuterVolumeSpecName: "kube-api-access-ml55h") pod "a6c37ff3-c6de-4852-97c8-58c28cbd3944" (UID: "a6c37ff3-c6de-4852-97c8-58c28cbd3944"). InnerVolumeSpecName "kube-api-access-ml55h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:01:01.467025 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.466993    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6c37ff3-c6de-4852-97c8-58c28cbd3944" (UID: "a6c37ff3-c6de-4852-97c8-58c28cbd3944"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:01:01.482602 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.482571    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6c37ff3-c6de-4852-97c8-58c28cbd3944" (UID: "a6c37ff3-c6de-4852-97c8-58c28cbd3944"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:01:01.520479 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.520428    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a6c37ff3-c6de-4852-97c8-58c28cbd3944" (UID: "a6c37ff3-c6de-4852-97c8-58c28cbd3944"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:01:01.565840 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.565791    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:01:01.565840 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.565835    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:01:01.565840 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.565844    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:01:01.565840 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.565853    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6c37ff3-c6de-4852-97c8-58c28cbd3944-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:01:01.566081 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.565861    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml55h\" (UniqueName: \"kubernetes.io/projected/a6c37ff3-c6de-4852-97c8-58c28cbd3944-kube-api-access-ml55h\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:01:01.566081 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.565870    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:01:01.566081 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.565877    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6c37ff3-c6de-4852-97c8-58c28cbd3944-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:01:01.603200 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.603162    2581 generic.go:356] "Generic (PLEG): container finished" podID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerID="f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61" exitCode=0
Jul 09 10:01:01.603331 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.603202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" event={"ID":"a6c37ff3-c6de-4852-97c8-58c28cbd3944","Type":"ContainerDied","Data":"f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61"}
Jul 09 10:01:01.603331 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.603259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6" event={"ID":"a6c37ff3-c6de-4852-97c8-58c28cbd3944","Type":"ContainerDied","Data":"55cee949878cd9794babdeb2914fc766118bb9bfebf49c9b8ad5afc14d190aa1"}
Jul 09 10:01:01.603331 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.603282    2581 scope.go:117] "RemoveContainer" containerID="f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61"
Jul 09 10:01:01.603331 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.603213    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"
Jul 09 10:01:01.612270 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.612251    2581 scope.go:117] "RemoveContainer" containerID="754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db"
Jul 09 10:01:01.619513 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.619494    2581 scope.go:117] "RemoveContainer" containerID="f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61"
Jul 09 10:01:01.619761 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:01:01.619742    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61\": container with ID starting with f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61 not found: ID does not exist" containerID="f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61"
Jul 09 10:01:01.619820 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.619768    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61"} err="failed to get container status \"f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61\": rpc error: code = NotFound desc = could not find container \"f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61\": container with ID starting with f6badaef03ad93b200c206f55747bf200a0de0b525f07dbecb7a441593070a61 not found: ID does not exist"
Jul 09 10:01:01.619820 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.619785    2581 scope.go:117] "RemoveContainer" containerID="754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db"
Jul 09 10:01:01.620071 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:01:01.620043    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db\": container with ID starting with 754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db not found: ID does not exist" containerID="754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db"
Jul 09 10:01:01.620110 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.620082    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db"} err="failed to get container status \"754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db\": rpc error: code = NotFound desc = could not find container \"754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db\": container with ID starting with 754c8fd4e149a399219addbee7c9453b2368cdf5db607c85368b5eb8370eb6db not found: ID does not exist"
Jul 09 10:01:01.625324 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.625304    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"]
Jul 09 10:01:01.628932 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:01.628913    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-7d748fc468f7cx6"]
Jul 09 10:01:02.228210 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:02.228179    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" path="/var/lib/kubelet/pods/a6c37ff3-c6de-4852-97c8-58c28cbd3944/volumes"
Jul 09 10:01:09.090439 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090411    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"]
Jul 09 10:01:09.090800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090661    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main"
Jul 09 10:01:09.090800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090672    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main"
Jul 09 10:01:09.090800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090685    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="storage-initializer"
Jul 09 10:01:09.090800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090691    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="storage-initializer"
Jul 09 10:01:09.090800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090708    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="storage-initializer"
Jul 09 10:01:09.090800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090713    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="storage-initializer"
Jul 09 10:01:09.090800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090721    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main"
Jul 09 10:01:09.090800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090726    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main"
Jul 09 10:01:09.090800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090766    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6c37ff3-c6de-4852-97c8-58c28cbd3944" containerName="main"
Jul 09 10:01:09.090800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.090775    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ed1aca-6f43-4140-ba64-76457b1c54ff" containerName="main"
Jul 09 10:01:09.093598 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.093582    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.095981 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.095957    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 10:01:09.095981 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.095965    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkhz8\""
Jul 09 10:01:09.096185 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.096095    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 10:01:09.096185 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.096124    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 09 10:01:09.105490 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.105471    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"]
Jul 09 10:01:09.118365 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.118336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.118469 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.118384    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2e48f8-1129-431a-af6c-84c1e49634d3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.118469 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.118432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.118469 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.118457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.118581 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.118475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.218999 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.218963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkxr\" (UniqueName: \"kubernetes.io/projected/ed2e48f8-1129-431a-af6c-84c1e49634d3-kube-api-access-5kkxr\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.219160 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.219019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.219160 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.219093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2e48f8-1129-431a-af6c-84c1e49634d3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.219160 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.219138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.219160 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.219160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.219298 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.219178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.219362 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.219344    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.219482 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.219466    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.219535 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.219497    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.221351 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.221324    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.221565 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.221547    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2e48f8-1129-431a-af6c-84c1e49634d3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.320024 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.319990    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkxr\" (UniqueName: \"kubernetes.io/projected/ed2e48f8-1129-431a-af6c-84c1e49634d3-kube-api-access-5kkxr\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:09.331113 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:09.331089    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkxr\" (UniqueName: \"kubernetes.io/projected/ed2e48f8-1129-431a-af6c-84c1e49634d3-kube-api-access-5kkxr\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:18.989275 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:18.989236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:18.994491 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:18.994454    2581 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:19.089937 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:19.089874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:19.090117 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:19.090064    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:19.093843 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:19.093823    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") DevicePath \"csi-a4decca2aaec13f4dfab50e9313a126be0c884fb9dbeffffba32cc7747037064\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:19.191653 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:19.191622    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/59c6c4e924c1e9af9a1fbe2af17a5efe6f7065d771aa791587657032d06c40d9/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:19.207331 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:19.207305    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:19.303209 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:19.303123    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:19.437491 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:19.437462    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"]
Jul 09 10:01:19.441050 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:01:19.441016    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded2e48f8_1129_431a_af6c_84c1e49634d3.slice/crio-0509e18b31508173a52942e56f78b319ab3c2f96920a0a9bd0e3c911ba8657c1 WatchSource:0}: Error finding container 0509e18b31508173a52942e56f78b319ab3c2f96920a0a9bd0e3c911ba8657c1: Status 404 returned error can't find the container with id 0509e18b31508173a52942e56f78b319ab3c2f96920a0a9bd0e3c911ba8657c1
Jul 09 10:01:19.672601 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:19.672501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" event={"ID":"ed2e48f8-1129-431a-af6c-84c1e49634d3","Type":"ContainerStarted","Data":"897048ac5fcf0f972744b145a309b4c589b4c87b5787cfc6af3ee286027c38a3"}
Jul 09 10:01:19.672601 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:19.672565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" event={"ID":"ed2e48f8-1129-431a-af6c-84c1e49634d3","Type":"ContainerStarted","Data":"0509e18b31508173a52942e56f78b319ab3c2f96920a0a9bd0e3c911ba8657c1"}
Jul 09 10:01:19.694415 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:19.694355    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podStartSLOduration=10.694337171 podStartE2EDuration="10.694337171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 10:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:01:19.693821837 +0000 UTC m=+1672.018784121" watchObservedRunningTime="2026-07-09 10:01:19.694337171 +0000 UTC m=+1672.019299457"
Jul 09 10:01:20.303515 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:20.303485    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:29.303688 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:29.303651    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:01:29.305268 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:29.305241    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:01:31.983585 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:31.983553    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"]
Jul 09 10:01:31.986649 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:31.986625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:31.989030 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:31.989009    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 10:01:31.989030 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:31.989021    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 09 10:01:32.000603 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.000576    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"]
Jul 09 10:01:32.089909 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.089849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-model-cache\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.090112 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.089963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-home\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.090112 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.089988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.090112 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.090009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-dshm\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.090112 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.090064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-tmp-dir\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.090112 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.090086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58acfa34-0297-4b40-91bf-589f32007c22-tls-certs\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.090112 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.090101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7nx\" (UniqueName: \"kubernetes.io/projected/58acfa34-0297-4b40-91bf-589f32007c22-kube-api-access-rq7nx\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191219 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-home\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191219 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191475 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-dshm\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191475 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-tmp-dir\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191475 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58acfa34-0297-4b40-91bf-589f32007c22-tls-certs\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191475 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7nx\" (UniqueName: \"kubernetes.io/projected/58acfa34-0297-4b40-91bf-589f32007c22-kube-api-access-rq7nx\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191693 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-model-cache\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191693 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191609    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-home\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191693 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191679    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191811 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191714    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-tmp-dir\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.191930 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.191882    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-model-cache\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.193594 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.193564    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-dshm\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.194145 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.194123    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58acfa34-0297-4b40-91bf-589f32007c22-tls-certs\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.200852 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.200828    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7nx\" (UniqueName: \"kubernetes.io/projected/58acfa34-0297-4b40-91bf-589f32007c22-kube-api-access-rq7nx\") pod \"custom-route-timeout-test-kserve-5cf8847bb6-mgmhx\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.296926 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.296812    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:32.438215 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.438190    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"]
Jul 09 10:01:32.440703 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:01:32.440673    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58acfa34_0297_4b40_91bf_589f32007c22.slice/crio-b6b2754003af7323a376e728bb53aad173ed5f658efe799f472ffc604b29cebb WatchSource:0}: Error finding container b6b2754003af7323a376e728bb53aad173ed5f658efe799f472ffc604b29cebb: Status 404 returned error can't find the container with id b6b2754003af7323a376e728bb53aad173ed5f658efe799f472ffc604b29cebb
Jul 09 10:01:32.723293 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.723248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" event={"ID":"58acfa34-0297-4b40-91bf-589f32007c22","Type":"ContainerStarted","Data":"a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd"}
Jul 09 10:01:32.723511 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:32.723319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" event={"ID":"58acfa34-0297-4b40-91bf-589f32007c22","Type":"ContainerStarted","Data":"b6b2754003af7323a376e728bb53aad173ed5f658efe799f472ffc604b29cebb"}
Jul 09 10:01:36.746781 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:36.746755    2581 generic.go:356] "Generic (PLEG): container finished" podID="58acfa34-0297-4b40-91bf-589f32007c22" containerID="a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd" exitCode=0
Jul 09 10:01:36.747085 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:36.746825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" event={"ID":"58acfa34-0297-4b40-91bf-589f32007c22","Type":"ContainerDied","Data":"a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd"}
Jul 09 10:01:37.753073 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:37.753032    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" event={"ID":"58acfa34-0297-4b40-91bf-589f32007c22","Type":"ContainerStarted","Data":"88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c"}
Jul 09 10:01:37.776286 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:37.776204    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podStartSLOduration=2.5519183610000002 podStartE2EDuration="6.776188026s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.224269665s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:01:37.775332056 +0000 UTC m=+1690.100294338" watchObservedRunningTime="2026-07-09 10:01:37.776188026 +0000 UTC m=+1690.101150363"
Jul 09 10:01:38.297353 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:38.297320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:39.304069 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:39.304021    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:01:42.297163 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:42.297128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:01:42.298570 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:42.298533    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:01:49.304344 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:49.304299    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:01:52.297968 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:52.297918    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:01:59.304256 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:01:59.304209    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:02:02.298390 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:02.298338    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:02:09.303852 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:09.303806    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:02:12.298323 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:12.298279    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:02:19.304257 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:19.304207    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:02:22.297807 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:22.297762    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:02:29.303604 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:29.303558    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:02:32.297815 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:32.297768    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:02:39.304258 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:39.304213    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:02:42.297351 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:42.297309    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:02:49.304134 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:49.304090    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:02:52.297444 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:52.297398    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:02:59.303849 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:02:59.303807    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:03:02.297872 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:02.297823    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:03:09.303651 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:09.303609    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:03:12.297866 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:12.297816    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:03:19.303836 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:19.303791    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:03:22.297406 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:22.297366    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:03:29.304651 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:29.304604    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": dial tcp 10.133.0.37:8000: connect: connection refused"
Jul 09 10:03:30.045194 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:30.045167    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:03:30.046409 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:30.046384    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:03:32.297375 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:32.297334    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:03:39.314143 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:39.314111    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:03:39.322567 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:39.322545    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:03:42.297490 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:42.297446    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:03:46.595559 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:46.595525    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"]
Jul 09 10:03:52.297626 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:03:52.297577    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:04:01.617714 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:01.617655    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" containerID="cri-o://897048ac5fcf0f972744b145a309b4c589b4c87b5787cfc6af3ee286027c38a3" gracePeriod=45
Jul 09 10:04:02.308955 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.308925    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:04:02.317742 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.317718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:04:02.354061 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.354027    2581 generic.go:356] "Generic (PLEG): container finished" podID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerID="897048ac5fcf0f972744b145a309b4c589b4c87b5787cfc6af3ee286027c38a3" exitCode=0
Jul 09 10:04:02.354207 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.354101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" event={"ID":"ed2e48f8-1129-431a-af6c-84c1e49634d3","Type":"ContainerDied","Data":"897048ac5fcf0f972744b145a309b4c589b4c87b5787cfc6af3ee286027c38a3"}
Jul 09 10:04:02.354207 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.354195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" event={"ID":"ed2e48f8-1129-431a-af6c-84c1e49634d3","Type":"ContainerDied","Data":"0509e18b31508173a52942e56f78b319ab3c2f96920a0a9bd0e3c911ba8657c1"}
Jul 09 10:04:02.354329 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.354224    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0509e18b31508173a52942e56f78b319ab3c2f96920a0a9bd0e3c911ba8657c1"
Jul 09 10:04:02.366420 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.366386    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:04:02.450596 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.450511    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-tmp-dir\") pod \"ed2e48f8-1129-431a-af6c-84c1e49634d3\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") "
Jul 09 10:04:02.450596 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.450545    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-dshm\") pod \"ed2e48f8-1129-431a-af6c-84c1e49634d3\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") "
Jul 09 10:04:02.450811 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.450652    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"ed2e48f8-1129-431a-af6c-84c1e49634d3\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") "
Jul 09 10:04:02.452724 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.452696    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-dshm" (OuterVolumeSpecName: "dshm") pod "ed2e48f8-1129-431a-af6c-84c1e49634d3" (UID: "ed2e48f8-1129-431a-af6c-84c1e49634d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:04:02.454899 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.454864    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522" (OuterVolumeSpecName: "kserve-pvc-source") pod "ed2e48f8-1129-431a-af6c-84c1e49634d3" (UID: "ed2e48f8-1129-431a-af6c-84c1e49634d3"). InnerVolumeSpecName "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 10:04:02.461555 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.461520    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed2e48f8-1129-431a-af6c-84c1e49634d3" (UID: "ed2e48f8-1129-431a-af6c-84c1e49634d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:04:02.551022 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.550991    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2e48f8-1129-431a-af6c-84c1e49634d3-tls-certs\") pod \"ed2e48f8-1129-431a-af6c-84c1e49634d3\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") "
Jul 09 10:04:02.551184 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.551032    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-home\") pod \"ed2e48f8-1129-431a-af6c-84c1e49634d3\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") "
Jul 09 10:04:02.551184 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.551054    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kkxr\" (UniqueName: \"kubernetes.io/projected/ed2e48f8-1129-431a-af6c-84c1e49634d3-kube-api-access-5kkxr\") pod \"ed2e48f8-1129-431a-af6c-84c1e49634d3\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") "
Jul 09 10:04:02.551184 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.551078    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-model-cache\") pod \"ed2e48f8-1129-431a-af6c-84c1e49634d3\" (UID: \"ed2e48f8-1129-431a-af6c-84c1e49634d3\") "
Jul 09 10:04:02.551303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.551286    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:04:02.551353 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.551310    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:04:02.551353 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.551349    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") on node \"ip-10-0-129-157.ec2.internal\" "
Jul 09 10:04:02.551447 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.551375    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed2e48f8-1129-431a-af6c-84c1e49634d3" (UID: "ed2e48f8-1129-431a-af6c-84c1e49634d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:04:02.551617 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.551578    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-home" (OuterVolumeSpecName: "home") pod "ed2e48f8-1129-431a-af6c-84c1e49634d3" (UID: "ed2e48f8-1129-431a-af6c-84c1e49634d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:04:02.553138 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.553116    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2e48f8-1129-431a-af6c-84c1e49634d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed2e48f8-1129-431a-af6c-84c1e49634d3" (UID: "ed2e48f8-1129-431a-af6c-84c1e49634d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:04:02.553290 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.553267    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2e48f8-1129-431a-af6c-84c1e49634d3-kube-api-access-5kkxr" (OuterVolumeSpecName: "kube-api-access-5kkxr") pod "ed2e48f8-1129-431a-af6c-84c1e49634d3" (UID: "ed2e48f8-1129-431a-af6c-84c1e49634d3"). InnerVolumeSpecName "kube-api-access-5kkxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:04:02.579586 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.579571    2581 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522") on node "ip-10-0-129-157.ec2.internal"
Jul 09 10:04:02.651637 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.651605    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"csi-a4decca2aaec13f4dfab50e9313a126be0c884fb9dbeffffba32cc7747037064\""
Jul 09 10:04:02.651637 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.651635    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2e48f8-1129-431a-af6c-84c1e49634d3-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:04:02.652061 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.651648    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:04:02.652061 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.651660    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kkxr\" (UniqueName: \"kubernetes.io/projected/ed2e48f8-1129-431a-af6c-84c1e49634d3-kube-api-access-5kkxr\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:04:02.652061 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:02.651672    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2e48f8-1129-431a-af6c-84c1e49634d3-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:04:03.315231 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:03.315185    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded"
Jul 09 10:04:03.358113 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:03.358086    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"
Jul 09 10:04:03.382440 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:03.382411    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"]
Jul 09 10:04:03.385077 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:03.385056    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9d5zs7r"]
Jul 09 10:04:04.228406 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:04.228374    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" path="/var/lib/kubelet/pods/ed2e48f8-1129-431a-af6c-84c1e49634d3/volumes"
Jul 09 10:04:11.042235 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.042199    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"]
Jul 09 10:04:11.042617 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.042485    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main"
Jul 09 10:04:11.042617 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.042496    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main"
Jul 09 10:04:11.042617 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.042554    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed2e48f8-1129-431a-af6c-84c1e49634d3" containerName="main"
Jul 09 10:04:11.045432 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.045416    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.052536 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.052517    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-8l48x\""
Jul 09 10:04:11.052843 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.052825    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 09 10:04:11.086437 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.086406    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"]
Jul 09 10:04:11.088583 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.088563    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.090750 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.090731    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"]
Jul 09 10:04:11.110868 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.110842    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"]
Jul 09 10:04:11.117883 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.117858    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.118016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.117915    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.118016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.117939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.118016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.117968    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99fpk\" (UniqueName: \"kubernetes.io/projected/470abb94-d8ca-4337-9092-baa05a5a9af7-kube-api-access-99fpk\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.118016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.117990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.118234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.118024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.118234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.118050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.118234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.118092    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.118234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.118183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwjs6\" (UniqueName: \"kubernetes.io/projected/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-kube-api-access-xwjs6\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.118234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.118211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.118445 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.118240    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.118445 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.118288    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470abb94-d8ca-4337-9092-baa05a5a9af7-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.219127 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwjs6\" (UniqueName: \"kubernetes.io/projected/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-kube-api-access-xwjs6\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.219127 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.219402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.219402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470abb94-d8ca-4337-9092-baa05a5a9af7-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.219402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.219402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.219402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219239    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.219402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99fpk\" (UniqueName: \"kubernetes.io/projected/470abb94-d8ca-4337-9092-baa05a5a9af7-kube-api-access-99fpk\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.219402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.219402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.219402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.219402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219386    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.219942 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219577    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.219942 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219619    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.219942 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219639    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.219942 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219700    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.219942 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.219931    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.220207 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.220039    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.221511 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.221481    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.221617 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.221602    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.222062 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.222043    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470abb94-d8ca-4337-9092-baa05a5a9af7-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:11.222197 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.222178    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.226836 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.226809    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwjs6\" (UniqueName: \"kubernetes.io/projected/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-kube-api-access-xwjs6\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:11.227018 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:11.226999    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99fpk\" (UniqueName: \"kubernetes.io/projected/470abb94-d8ca-4337-9092-baa05a5a9af7-kube-api-access-99fpk\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:12.430174 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.430136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:12.435873 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.435848    2581 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:12.531631 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.531596    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:12.531777 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.531706    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:12.534341 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.534322    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") DevicePath \"csi-a4decca2aaec13f4dfab50e9313a126be0c884fb9dbeffffba32cc7747037064\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:12.598791 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.598763    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/59c6c4e924c1e9af9a1fbe2af17a5efe6f7065d771aa791587657032d06c40d9/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:12.614463 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.614437    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:12.632832 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.632794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:12.633014 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.632929    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") DevicePath \"csi-a4decca2aaec13f4dfab50e9313a126be0c884fb9dbeffffba32cc7747037064\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:12.635614 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.635582    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") DevicePath \"csi-a4decca2aaec13f4dfab50e9313a126be0c884fb9dbeffffba32cc7747037064\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:12.647234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.647208    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:12.854133 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.854089    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:12.898789 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:12.898755    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:13.002605 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:13.002574    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"]
Jul 09 10:04:13.006126 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:04:13.006096    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470abb94_d8ca_4337_9092_baa05a5a9af7.slice/crio-5a005f62a5adc286cf5dfc02bd321f9855f36895dacd0c35ce0214fa0658da4e WatchSource:0}: Error finding container 5a005f62a5adc286cf5dfc02bd321f9855f36895dacd0c35ce0214fa0658da4e: Status 404 returned error can't find the container with id 5a005f62a5adc286cf5dfc02bd321f9855f36895dacd0c35ce0214fa0658da4e
Jul 09 10:04:13.008285 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:13.008262    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 10:04:13.038710 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:13.038640    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"]
Jul 09 10:04:13.040566 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:04:13.040539    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551532a7_a8e9_4983_a7ed_5ee993b4ef1c.slice/crio-206527e8bee2e7f72eabca8a2e69b4d68ff4b495c1b6f775a96e6392a8ac08f7 WatchSource:0}: Error finding container 206527e8bee2e7f72eabca8a2e69b4d68ff4b495c1b6f775a96e6392a8ac08f7: Status 404 returned error can't find the container with id 206527e8bee2e7f72eabca8a2e69b4d68ff4b495c1b6f775a96e6392a8ac08f7
Jul 09 10:04:13.398252 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:13.398211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" event={"ID":"551532a7-a8e9-4983-a7ed-5ee993b4ef1c","Type":"ContainerStarted","Data":"7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea"}
Jul 09 10:04:13.398435 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:13.398268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" event={"ID":"551532a7-a8e9-4983-a7ed-5ee993b4ef1c","Type":"ContainerStarted","Data":"206527e8bee2e7f72eabca8a2e69b4d68ff4b495c1b6f775a96e6392a8ac08f7"}
Jul 09 10:04:13.399416 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:13.399394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" event={"ID":"470abb94-d8ca-4337-9092-baa05a5a9af7","Type":"ContainerStarted","Data":"5a005f62a5adc286cf5dfc02bd321f9855f36895dacd0c35ce0214fa0658da4e"}
Jul 09 10:04:13.420288 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:13.420245    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podStartSLOduration=2.420228679 podStartE2EDuration="2.420228679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 10:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:04:13.418516962 +0000 UTC m=+1845.743479237" watchObservedRunningTime="2026-07-09 10:04:13.420228679 +0000 UTC m=+1845.745191009"
Jul 09 10:04:14.405285 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:14.405242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" event={"ID":"470abb94-d8ca-4337-9092-baa05a5a9af7","Type":"ContainerStarted","Data":"8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb"}
Jul 09 10:04:14.405753 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:14.405445    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:15.412275 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:15.412236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" event={"ID":"470abb94-d8ca-4337-9092-baa05a5a9af7","Type":"ContainerStarted","Data":"378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2"}
Jul 09 10:04:15.451010 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:15.450957    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podStartSLOduration=3.230994022 podStartE2EDuration="4.450940392s" totalImagesPullingTime="1.21994637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 10:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:04:15.450125228 +0000 UTC m=+1847.775087547" watchObservedRunningTime="2026-07-09 10:04:15.450940392 +0000 UTC m=+1847.775902676"
Jul 09 10:04:22.854438 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:22.854400    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:22.854930 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:22.854464    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:22.855875 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:22.855843    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:04:22.899648 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:22.899620    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:22.899807 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:22.899775    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:04:22.901031 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:22.901005    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:04:32.854820 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:32.854772    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:04:32.880997 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:32.880966    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:04:32.899240 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:32.899199    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:04:42.854988 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:42.854937    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:04:42.899954 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:42.899912    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:04:52.854814 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:52.854767    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:04:52.899463 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:04:52.899428    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:05:02.855017 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:02.854972    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:05:02.899972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:02.899925    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:05:12.854796 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:12.854747    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:05:12.899467 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:12.899420    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:05:22.855251 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:22.855204    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:05:22.899703 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:22.899669    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:05:32.855291 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:32.855247    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:05:32.899809 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:32.899776    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:05:42.855318 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:42.855267    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:05:42.900110 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:42.900073    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:05:52.855378 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:52.855337    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:05:52.899253 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:05:52.899210    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:06:02.855301 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:02.855252    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:06:02.900152 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:02.900114    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:06:12.855014 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:12.854957    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:06:12.899764 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:12.899717    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:06:22.855157 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:22.855053    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:06:22.899121 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:22.899088    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:06:32.855041 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:32.855003    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:06:32.899799 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:32.899767    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:06:42.854479 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:42.854439    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8001/health\": dial tcp 10.133.0.39:8001: connect: connection refused"
Jul 09 10:06:42.899685 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:42.899638    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": dial tcp 10.133.0.40:8000: connect: connection refused"
Jul 09 10:06:52.869122 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:52.869090    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:06:52.880881 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:52.880855    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:06:52.908656 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:52.908628    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:06:52.916694 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:52.916670    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:06:58.387718 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:58.387679    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"]
Jul 09 10:06:58.389401 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:06:58.389377    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"]
Jul 09 10:07:13.421459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:13.421396    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main" containerID="cri-o://7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea" gracePeriod=45
Jul 09 10:07:13.421877 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:13.421732    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main" containerID="cri-o://378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2" gracePeriod=45
Jul 09 10:07:14.237927 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.237874    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="llm-d-routing-sidecar" containerID="cri-o://8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb" gracePeriod=45
Jul 09 10:07:14.372964 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.372935    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:07:14.411609 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.411585    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-home\") pod \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") "
Jul 09 10:07:14.411741 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.411617    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-dshm\") pod \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") "
Jul 09 10:07:14.411741 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.411646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tmp-dir\") pod \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") "
Jul 09 10:07:14.411741 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.411661    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tls-certs\") pod \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") "
Jul 09 10:07:14.411741 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.411695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwjs6\" (UniqueName: \"kubernetes.io/projected/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-kube-api-access-xwjs6\") pod \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") "
Jul 09 10:07:14.411998 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.411838    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") "
Jul 09 10:07:14.411998 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.411884    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-model-cache\") pod \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\" (UID: \"551532a7-a8e9-4983-a7ed-5ee993b4ef1c\") "
Jul 09 10:07:14.412288 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.412261    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-home" (OuterVolumeSpecName: "home") pod "551532a7-a8e9-4983-a7ed-5ee993b4ef1c" (UID: "551532a7-a8e9-4983-a7ed-5ee993b4ef1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:07:14.412546 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.412444    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-model-cache" (OuterVolumeSpecName: "model-cache") pod "551532a7-a8e9-4983-a7ed-5ee993b4ef1c" (UID: "551532a7-a8e9-4983-a7ed-5ee993b4ef1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:07:14.414665 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.414625    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "551532a7-a8e9-4983-a7ed-5ee993b4ef1c" (UID: "551532a7-a8e9-4983-a7ed-5ee993b4ef1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:07:14.414794 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.414687    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-dshm" (OuterVolumeSpecName: "dshm") pod "551532a7-a8e9-4983-a7ed-5ee993b4ef1c" (UID: "551532a7-a8e9-4983-a7ed-5ee993b4ef1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:07:14.414995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.414968    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-kube-api-access-xwjs6" (OuterVolumeSpecName: "kube-api-access-xwjs6") pod "551532a7-a8e9-4983-a7ed-5ee993b4ef1c" (UID: "551532a7-a8e9-4983-a7ed-5ee993b4ef1c"). InnerVolumeSpecName "kube-api-access-xwjs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:07:14.416521 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.416500    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522" (OuterVolumeSpecName: "kserve-pvc-source") pod "551532a7-a8e9-4983-a7ed-5ee993b4ef1c" (UID: "551532a7-a8e9-4983-a7ed-5ee993b4ef1c"). InnerVolumeSpecName "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 10:07:14.424244 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.424218    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "551532a7-a8e9-4983-a7ed-5ee993b4ef1c" (UID: "551532a7-a8e9-4983-a7ed-5ee993b4ef1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:07:14.477248 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.477222    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:07:14.512700 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.512675    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-model-cache\") pod \"470abb94-d8ca-4337-9092-baa05a5a9af7\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") "
Jul 09 10:07:14.512826 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.512770    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"470abb94-d8ca-4337-9092-baa05a5a9af7\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") "
Jul 09 10:07:14.512826 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.512793    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99fpk\" (UniqueName: \"kubernetes.io/projected/470abb94-d8ca-4337-9092-baa05a5a9af7-kube-api-access-99fpk\") pod \"470abb94-d8ca-4337-9092-baa05a5a9af7\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") "
Jul 09 10:07:14.512956 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.512833    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-home\") pod \"470abb94-d8ca-4337-9092-baa05a5a9af7\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") "
Jul 09 10:07:14.512956 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.512867    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-tmp-dir\") pod \"470abb94-d8ca-4337-9092-baa05a5a9af7\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") "
Jul 09 10:07:14.512956 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.512906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470abb94-d8ca-4337-9092-baa05a5a9af7-tls-certs\") pod \"470abb94-d8ca-4337-9092-baa05a5a9af7\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") "
Jul 09 10:07:14.512956 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.512954    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-dshm\") pod \"470abb94-d8ca-4337-9092-baa05a5a9af7\" (UID: \"470abb94-d8ca-4337-9092-baa05a5a9af7\") "
Jul 09 10:07:14.513153 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.512952    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-model-cache" (OuterVolumeSpecName: "model-cache") pod "470abb94-d8ca-4337-9092-baa05a5a9af7" (UID: "470abb94-d8ca-4337-9092-baa05a5a9af7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:07:14.513206 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.513170    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.513206 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.513188    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.513206 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.513204    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.513389 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.513217    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.513389 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.513231    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.513389 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.513243    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.513389 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.513257    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwjs6\" (UniqueName: \"kubernetes.io/projected/551532a7-a8e9-4983-a7ed-5ee993b4ef1c-kube-api-access-xwjs6\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.513589 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.513530    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-home" (OuterVolumeSpecName: "home") pod "470abb94-d8ca-4337-9092-baa05a5a9af7" (UID: "470abb94-d8ca-4337-9092-baa05a5a9af7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:07:14.515719 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.515671    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-dshm" (OuterVolumeSpecName: "dshm") pod "470abb94-d8ca-4337-9092-baa05a5a9af7" (UID: "470abb94-d8ca-4337-9092-baa05a5a9af7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:07:14.515829 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.515804    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470abb94-d8ca-4337-9092-baa05a5a9af7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "470abb94-d8ca-4337-9092-baa05a5a9af7" (UID: "470abb94-d8ca-4337-9092-baa05a5a9af7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:07:14.516065 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.516039    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470abb94-d8ca-4337-9092-baa05a5a9af7-kube-api-access-99fpk" (OuterVolumeSpecName: "kube-api-access-99fpk") pod "470abb94-d8ca-4337-9092-baa05a5a9af7" (UID: "470abb94-d8ca-4337-9092-baa05a5a9af7"). InnerVolumeSpecName "kube-api-access-99fpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:07:14.516180 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.516169    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522" (OuterVolumeSpecName: "kserve-pvc-source") pod "470abb94-d8ca-4337-9092-baa05a5a9af7" (UID: "470abb94-d8ca-4337-9092-baa05a5a9af7"). InnerVolumeSpecName "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 10:07:14.524869 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.524844    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "470abb94-d8ca-4337-9092-baa05a5a9af7" (UID: "470abb94-d8ca-4337-9092-baa05a5a9af7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:07:14.614585 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.614550    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") on node \"ip-10-0-129-157.ec2.internal\" "
Jul 09 10:07:14.614585 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.614580    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99fpk\" (UniqueName: \"kubernetes.io/projected/470abb94-d8ca-4337-9092-baa05a5a9af7-kube-api-access-99fpk\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.614585 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.614590    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.614825 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.614599    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.614825 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.614609    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/470abb94-d8ca-4337-9092-baa05a5a9af7-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.614825 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.614616    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/470abb94-d8ca-4337-9092-baa05a5a9af7-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:07:14.641949 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.641928    2581 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522") on node "ip-10-0-129-157.ec2.internal"
Jul 09 10:07:14.715487 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:14.715460    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"csi-a4decca2aaec13f4dfab50e9313a126be0c884fb9dbeffffba32cc7747037064\""
Jul 09 10:07:15.170034 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.169997    2581 generic.go:356] "Generic (PLEG): container finished" podID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerID="7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea" exitCode=0
Jul 09 10:07:15.170190 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.170082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" event={"ID":"551532a7-a8e9-4983-a7ed-5ee993b4ef1c","Type":"ContainerDied","Data":"7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea"}
Jul 09 10:07:15.170190 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.170115    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"
Jul 09 10:07:15.170190 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.170174    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2" event={"ID":"551532a7-a8e9-4983-a7ed-5ee993b4ef1c","Type":"ContainerDied","Data":"206527e8bee2e7f72eabca8a2e69b4d68ff4b495c1b6f775a96e6392a8ac08f7"}
Jul 09 10:07:15.170379 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.170205    2581 scope.go:117] "RemoveContainer" containerID="7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea"
Jul 09 10:07:15.171847 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.171826    2581 generic.go:356] "Generic (PLEG): container finished" podID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerID="378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2" exitCode=0
Jul 09 10:07:15.172009 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.171849    2581 generic.go:356] "Generic (PLEG): container finished" podID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerID="8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb" exitCode=0
Jul 09 10:07:15.172009 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.171917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" event={"ID":"470abb94-d8ca-4337-9092-baa05a5a9af7","Type":"ContainerDied","Data":"378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2"}
Jul 09 10:07:15.172009 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.171974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" event={"ID":"470abb94-d8ca-4337-9092-baa05a5a9af7","Type":"ContainerDied","Data":"8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb"}
Jul 09 10:07:15.172009 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.171999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4" event={"ID":"470abb94-d8ca-4337-9092-baa05a5a9af7","Type":"ContainerDied","Data":"5a005f62a5adc286cf5dfc02bd321f9855f36895dacd0c35ce0214fa0658da4e"}
Jul 09 10:07:15.172174 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.172091    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"
Jul 09 10:07:15.180199 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.180182    2581 scope.go:117] "RemoveContainer" containerID="7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea"
Jul 09 10:07:15.180637 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:07:15.180453    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea\": container with ID starting with 7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea not found: ID does not exist" containerID="7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea"
Jul 09 10:07:15.180637 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.180485    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea"} err="failed to get container status \"7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea\": rpc error: code = NotFound desc = could not find container \"7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea\": container with ID starting with 7f3039df2f0f66368c49930f05f58996f8983e5057f858cce33963fb5f2f02ea not found: ID does not exist"
Jul 09 10:07:15.180637 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.180506    2581 scope.go:117] "RemoveContainer" containerID="378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2"
Jul 09 10:07:15.188443 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.188428    2581 scope.go:117] "RemoveContainer" containerID="8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb"
Jul 09 10:07:15.196740 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.196713    2581 scope.go:117] "RemoveContainer" containerID="378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2"
Jul 09 10:07:15.197161 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:07:15.197137    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2\": container with ID starting with 378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2 not found: ID does not exist" containerID="378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2"
Jul 09 10:07:15.197315 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.197171    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2"} err="failed to get container status \"378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2\": rpc error: code = NotFound desc = could not find container \"378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2\": container with ID starting with 378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2 not found: ID does not exist"
Jul 09 10:07:15.197315 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.197195    2581 scope.go:117] "RemoveContainer" containerID="8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb"
Jul 09 10:07:15.197715 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:07:15.197492    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb\": container with ID starting with 8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb not found: ID does not exist" containerID="8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb"
Jul 09 10:07:15.197715 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.197538    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb"} err="failed to get container status \"8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb\": rpc error: code = NotFound desc = could not find container \"8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb\": container with ID starting with 8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb not found: ID does not exist"
Jul 09 10:07:15.197715 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.197557    2581 scope.go:117] "RemoveContainer" containerID="378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2"
Jul 09 10:07:15.198303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.197792    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2"} err="failed to get container status \"378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2\": rpc error: code = NotFound desc = could not find container \"378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2\": container with ID starting with 378f30d6200250e714ad5f282757897546d739ca3806e8fbe88b71db072a11f2 not found: ID does not exist"
Jul 09 10:07:15.198303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.197816    2581 scope.go:117] "RemoveContainer" containerID="8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb"
Jul 09 10:07:15.198303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.198069    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb"} err="failed to get container status \"8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb\": rpc error: code = NotFound desc = could not find container \"8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb\": container with ID starting with 8b85df6906c2f684ba1531a5f0b13247adeb85304316881ac675a71f61666deb not found: ID does not exist"
Jul 09 10:07:15.198558 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.198540    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"]
Jul 09 10:07:15.202592 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.202571    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-rq4x4"]
Jul 09 10:07:15.212938 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.212916    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"]
Jul 09 10:07:15.219120 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:15.219087    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79f2mv2"]
Jul 09 10:07:16.228401 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:16.228367    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" path="/var/lib/kubelet/pods/470abb94-d8ca-4337-9092-baa05a5a9af7/volumes"
Jul 09 10:07:16.228847 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:16.228832    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" path="/var/lib/kubelet/pods/551532a7-a8e9-4983-a7ed-5ee993b4ef1c/volumes"
Jul 09 10:07:28.283635 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:28.283605    2581 scope.go:117] "RemoveContainer" containerID="897048ac5fcf0f972744b145a309b4c589b4c87b5787cfc6af3ee286027c38a3"
Jul 09 10:07:36.864963 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.864921    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"]
Jul 09 10:07:36.865382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.865228    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="llm-d-routing-sidecar"
Jul 09 10:07:36.865382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.865244    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="llm-d-routing-sidecar"
Jul 09 10:07:36.865382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.865265    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main"
Jul 09 10:07:36.865382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.865270    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main"
Jul 09 10:07:36.865382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.865284    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main"
Jul 09 10:07:36.865382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.865290    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main"
Jul 09 10:07:36.865382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.865345    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="main"
Jul 09 10:07:36.865382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.865354    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="470abb94-d8ca-4337-9092-baa05a5a9af7" containerName="llm-d-routing-sidecar"
Jul 09 10:07:36.865382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.865362    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="551532a7-a8e9-4983-a7ed-5ee993b4ef1c" containerName="main"
Jul 09 10:07:36.867180 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.867163    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 10:07:36.867331 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.867309    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:36.869498 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.869480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:36.869823 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.869804    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 09 10:07:36.869926 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.869901    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e794-epp-sa-dockercfg-tbgrr\""
Jul 09 10:07:36.871740 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.871721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-g8fq2\""
Jul 09 10:07:36.882608 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.882586    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"]
Jul 09 10:07:36.889373 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.889354    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 10:07:36.915289 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.915257    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 10:07:36.917519 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.917499    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:36.929557 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.929537    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 10:07:36.994403 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:36.994403 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8307803a-aaf8-403d-bcb7-07fe84c92555-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:36.994598 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9xmb\" (UniqueName: \"kubernetes.io/projected/8307803a-aaf8-403d-bcb7-07fe84c92555-kube-api-access-k9xmb\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:36.994598 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmrj6\" (UniqueName: \"kubernetes.io/projected/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-kube-api-access-tmrj6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:36.994598 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70322401-e103-4e95-8134-0c36df44de20-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:36.994598 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:36.994598 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:36.994598 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:36.994792 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8307803a-aaf8-403d-bcb7-07fe84c92555-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:36.994792 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:36.994792 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:36.994792 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994700    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:36.994792 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9b97\" (UniqueName: \"kubernetes.io/projected/70322401-e103-4e95-8134-0c36df44de20-kube-api-access-n9b97\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:36.994792 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:36.995003 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:36.994809    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.095667 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.095633    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.095667 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.095668    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8307803a-aaf8-403d-bcb7-07fe84c92555-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:37.095879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.095685    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9xmb\" (UniqueName: \"kubernetes.io/projected/8307803a-aaf8-403d-bcb7-07fe84c92555-kube-api-access-k9xmb\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:37.095879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.095717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmrj6\" (UniqueName: \"kubernetes.io/projected/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-kube-api-access-tmrj6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.095879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.095786    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70322401-e103-4e95-8134-0c36df44de20-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.095879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.095829    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.095879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.095861    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.096183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.095912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.096183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.095942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8307803a-aaf8-403d-bcb7-07fe84c92555-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:37.096183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096067    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.096183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096109    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.096183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096131    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8307803a-aaf8-403d-bcb7-07fe84c92555-tokenizer-uds\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:37.096183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096147    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.096494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.096494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9b97\" (UniqueName: \"kubernetes.io/projected/70322401-e103-4e95-8134-0c36df44de20-kube-api-access-n9b97\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.096494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096243    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.096494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.096494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.096494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096490    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.096816 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096555    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.096816 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096656    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.096816 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.096749    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.098278 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.098249    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.098453 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.098423    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.098574 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.098556    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8307803a-aaf8-403d-bcb7-07fe84c92555-tls-certs\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:37.098701 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.098682    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.098770 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.098759    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70322401-e103-4e95-8134-0c36df44de20-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.103905 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.103870    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmrj6\" (UniqueName: \"kubernetes.io/projected/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-kube-api-access-tmrj6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:37.104452 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.104429    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9b97\" (UniqueName: \"kubernetes.io/projected/70322401-e103-4e95-8134-0c36df44de20-kube-api-access-n9b97\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:37.104745 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.104723    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9xmb\" (UniqueName: \"kubernetes.io/projected/8307803a-aaf8-403d-bcb7-07fe84c92555-kube-api-access-k9xmb\") pod \"llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") " pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:37.178041 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.177964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:37.310143 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:37.310108    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"]
Jul 09 10:07:37.311518 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:07:37.311492    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8307803a_aaf8_403d_bcb7_07fe84c92555.slice/crio-946be4d600dfe7c4a0a09cbf56f40145052827b3df847f09c82b8749561443a1 WatchSource:0}: Error finding container 946be4d600dfe7c4a0a09cbf56f40145052827b3df847f09c82b8749561443a1: Status 404 returned error can't find the container with id 946be4d600dfe7c4a0a09cbf56f40145052827b3df847f09c82b8749561443a1
Jul 09 10:07:38.258701 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:38.258669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj" event={"ID":"8307803a-aaf8-403d-bcb7-07fe84c92555","Type":"ContainerStarted","Data":"e1e775f7ab465e13e5ff0a79ec350b6a7132ad91cdf9fe24e0e22fcf9e2ee643"}
Jul 09 10:07:38.259115 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:38.258714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj" event={"ID":"8307803a-aaf8-403d-bcb7-07fe84c92555","Type":"ContainerStarted","Data":"946be4d600dfe7c4a0a09cbf56f40145052827b3df847f09c82b8749561443a1"}
Jul 09 10:07:38.259115 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:38.258815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:07:38.280418 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:38.280374    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj" podStartSLOduration=2.280357042 podStartE2EDuration="2.280357042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 10:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:07:38.279264134 +0000 UTC m=+2050.604226430" watchObservedRunningTime="2026-07-09 10:07:38.280357042 +0000 UTC m=+2050.605319326"
Jul 09 10:07:46.480210 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.480171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:46.485607 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.485577    2581 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:46.586709 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.582179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:46.586709 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.582423    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:46.586709 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.585368    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") DevicePath \"csi-a4decca2aaec13f4dfab50e9313a126be0c884fb9dbeffffba32cc7747037064\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:46.677073 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.677039    2581 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/59c6c4e924c1e9af9a1fbe2af17a5efe6f7065d771aa791587657032d06c40d9/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:46.688265 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.688238    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:46.784109 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.784021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:46.784258 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.784158    2581 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") DevicePath \"csi-a4decca2aaec13f4dfab50e9313a126be0c884fb9dbeffffba32cc7747037064\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:46.787327 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.787305    2581 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") DevicePath \"csi-a4decca2aaec13f4dfab50e9313a126be0c884fb9dbeffffba32cc7747037064\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:46.797878 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.797850    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:46.826988 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.826955    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:07:46.958297 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:46.958260    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 10:07:46.961139 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:07:46.961113    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb12740_c438_4ed8_b9a1_d11aa0a2ba21.slice/crio-16ce2a070d8d8a3f59382c51e3f64d161ba41101d95cb660a225475b5cf84b33 WatchSource:0}: Error finding container 16ce2a070d8d8a3f59382c51e3f64d161ba41101d95cb660a225475b5cf84b33: Status 404 returned error can't find the container with id 16ce2a070d8d8a3f59382c51e3f64d161ba41101d95cb660a225475b5cf84b33
Jul 09 10:07:47.084071 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:47.084035    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:47.234626 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:47.234594    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 10:07:47.237666 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:07:47.237641    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70322401_e103_4e95_8134_0c36df44de20.slice/crio-612fb59b8ef32bacf3db79c1ea4cc9138a11fa1a442dda0fc34c6e1ba77df405 WatchSource:0}: Error finding container 612fb59b8ef32bacf3db79c1ea4cc9138a11fa1a442dda0fc34c6e1ba77df405: Status 404 returned error can't find the container with id 612fb59b8ef32bacf3db79c1ea4cc9138a11fa1a442dda0fc34c6e1ba77df405
Jul 09 10:07:47.295237 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:47.295211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"70322401-e103-4e95-8134-0c36df44de20","Type":"ContainerStarted","Data":"612fb59b8ef32bacf3db79c1ea4cc9138a11fa1a442dda0fc34c6e1ba77df405"}
Jul 09 10:07:47.296983 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:47.296926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21","Type":"ContainerStarted","Data":"40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac"}
Jul 09 10:07:47.296983 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:47.296981    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21","Type":"ContainerStarted","Data":"16ce2a070d8d8a3f59382c51e3f64d161ba41101d95cb660a225475b5cf84b33"}
Jul 09 10:07:47.317907 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:47.317823    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=11.317802117 podStartE2EDuration="11.317802117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 10:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:07:47.317699487 +0000 UTC m=+2059.642661773" watchObservedRunningTime="2026-07-09 10:07:47.317802117 +0000 UTC m=+2059.642764402"
Jul 09 10:07:48.302731 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:48.302692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"70322401-e103-4e95-8134-0c36df44de20","Type":"ContainerStarted","Data":"29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6"}
Jul 09 10:07:48.327540 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:48.327483    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=12.327462238 podStartE2EDuration="12.327462238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 10:07:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:07:48.325407248 +0000 UTC m=+2060.650369533" watchObservedRunningTime="2026-07-09 10:07:48.327462238 +0000 UTC m=+2060.652424522"
Jul 09 10:07:49.084290 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:49.084254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:57.085118 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:57.085018    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:07:57.086612 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:07:57.086586    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:08:07.085458 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:07.085412    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:08:09.265021 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:09.264993    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:08:17.085574 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:17.085525    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:08:27.085457 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:27.085412    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:08:30.086786 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:30.086748    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:08:30.089911 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:30.089861    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:08:35.226301 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:35.226270    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"]
Jul 09 10:08:37.085402 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:37.085351    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:08:47.085455 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:47.085410    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:08:50.253004 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:50.252962    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" containerID="cri-o://88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c" gracePeriod=45
Jul 09 10:08:51.310240 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.310199    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/health\": dial tcp 10.133.0.38:8000: connect: connection refused"
Jul 09 10:08:51.517477 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.517402    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:08:51.592285 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.592256    2581 generic.go:356] "Generic (PLEG): container finished" podID="58acfa34-0297-4b40-91bf-589f32007c22" containerID="88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c" exitCode=0
Jul 09 10:08:51.592473 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.592294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" event={"ID":"58acfa34-0297-4b40-91bf-589f32007c22","Type":"ContainerDied","Data":"88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c"}
Jul 09 10:08:51.592473 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.592315    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"
Jul 09 10:08:51.592473 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.592348    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx" event={"ID":"58acfa34-0297-4b40-91bf-589f32007c22","Type":"ContainerDied","Data":"b6b2754003af7323a376e728bb53aad173ed5f658efe799f472ffc604b29cebb"}
Jul 09 10:08:51.592473 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.592376    2581 scope.go:117] "RemoveContainer" containerID="88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c"
Jul 09 10:08:51.601070 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.601040    2581 scope.go:117] "RemoveContainer" containerID="a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd"
Jul 09 10:08:51.610177 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.610156    2581 scope.go:117] "RemoveContainer" containerID="88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c"
Jul 09 10:08:51.610416 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:08:51.610399    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c\": container with ID starting with 88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c not found: ID does not exist" containerID="88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c"
Jul 09 10:08:51.610474 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.610423    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c"} err="failed to get container status \"88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c\": rpc error: code = NotFound desc = could not find container \"88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c\": container with ID starting with 88246b52a99d4284179005a28403227b8f3274454aa8f9732cb1c99d2432b89c not found: ID does not exist"
Jul 09 10:08:51.610474 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.610438    2581 scope.go:117] "RemoveContainer" containerID="a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd"
Jul 09 10:08:51.610682 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:08:51.610662    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd\": container with ID starting with a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd not found: ID does not exist" containerID="a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd"
Jul 09 10:08:51.610734 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.610687    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd"} err="failed to get container status \"a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd\": rpc error: code = NotFound desc = could not find container \"a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd\": container with ID starting with a4fc963374d016f893a60eb9f784e845382de92c0e798dc867c582105dd65ffd not found: ID does not exist"
Jul 09 10:08:51.704868 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.704835    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58acfa34-0297-4b40-91bf-589f32007c22-tls-certs\") pod \"58acfa34-0297-4b40-91bf-589f32007c22\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") "
Jul 09 10:08:51.705060 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.704877    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-home\") pod \"58acfa34-0297-4b40-91bf-589f32007c22\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") "
Jul 09 10:08:51.705060 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.704932    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-kserve-provision-location\") pod \"58acfa34-0297-4b40-91bf-589f32007c22\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") "
Jul 09 10:08:51.705060 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.705043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-tmp-dir\") pod \"58acfa34-0297-4b40-91bf-589f32007c22\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") "
Jul 09 10:08:51.705234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.705091    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-model-cache\") pod \"58acfa34-0297-4b40-91bf-589f32007c22\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") "
Jul 09 10:08:51.705234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.705125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rq7nx\" (UniqueName: \"kubernetes.io/projected/58acfa34-0297-4b40-91bf-589f32007c22-kube-api-access-rq7nx\") pod \"58acfa34-0297-4b40-91bf-589f32007c22\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") "
Jul 09 10:08:51.705234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.705165    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-dshm\") pod \"58acfa34-0297-4b40-91bf-589f32007c22\" (UID: \"58acfa34-0297-4b40-91bf-589f32007c22\") "
Jul 09 10:08:51.705494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.705465    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-model-cache" (OuterVolumeSpecName: "model-cache") pod "58acfa34-0297-4b40-91bf-589f32007c22" (UID: "58acfa34-0297-4b40-91bf-589f32007c22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:08:51.705635 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.705606    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-home" (OuterVolumeSpecName: "home") pod "58acfa34-0297-4b40-91bf-589f32007c22" (UID: "58acfa34-0297-4b40-91bf-589f32007c22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:08:51.707653 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.707628    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58acfa34-0297-4b40-91bf-589f32007c22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58acfa34-0297-4b40-91bf-589f32007c22" (UID: "58acfa34-0297-4b40-91bf-589f32007c22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:08:51.707803 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.707780    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-dshm" (OuterVolumeSpecName: "dshm") pod "58acfa34-0297-4b40-91bf-589f32007c22" (UID: "58acfa34-0297-4b40-91bf-589f32007c22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:08:51.707803 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.707791    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58acfa34-0297-4b40-91bf-589f32007c22-kube-api-access-rq7nx" (OuterVolumeSpecName: "kube-api-access-rq7nx") pod "58acfa34-0297-4b40-91bf-589f32007c22" (UID: "58acfa34-0297-4b40-91bf-589f32007c22"). InnerVolumeSpecName "kube-api-access-rq7nx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:08:51.722383 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.722355    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58acfa34-0297-4b40-91bf-589f32007c22" (UID: "58acfa34-0297-4b40-91bf-589f32007c22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:08:51.767528 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.767429    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58acfa34-0297-4b40-91bf-589f32007c22" (UID: "58acfa34-0297-4b40-91bf-589f32007c22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:08:51.806557 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.806520    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:08:51.806557 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.806551    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58acfa34-0297-4b40-91bf-589f32007c22-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:08:51.806557 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.806561    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:08:51.806776 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.806570    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:08:51.806776 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.806579    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:08:51.806776 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.806588    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58acfa34-0297-4b40-91bf-589f32007c22-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:08:51.806776 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.806597    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq7nx\" (UniqueName: \"kubernetes.io/projected/58acfa34-0297-4b40-91bf-589f32007c22-kube-api-access-rq7nx\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:08:51.916948 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.916918    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"]
Jul 09 10:08:51.922831 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:51.922802    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-5cf8847bb6-mgmhx"]
Jul 09 10:08:52.228098 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:52.228068    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58acfa34-0297-4b40-91bf-589f32007c22" path="/var/lib/kubelet/pods/58acfa34-0297-4b40-91bf-589f32007c22/volumes"
Jul 09 10:08:57.085237 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:08:57.085187    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:09:07.085459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.085407    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:09:07.687470 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.687439    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"]
Jul 09 10:09:07.687729 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.687717    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main"
Jul 09 10:09:07.687774 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.687731    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main"
Jul 09 10:09:07.687774 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.687745    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="storage-initializer"
Jul 09 10:09:07.687774 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.687751    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="storage-initializer"
Jul 09 10:09:07.687871 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.687801    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="58acfa34-0297-4b40-91bf-589f32007c22" containerName="main"
Jul 09 10:09:07.690715 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.690696    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.693123 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.693086    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 09 10:09:07.693254 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.693129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 10:09:07.705134 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.705110    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"]
Jul 09 10:09:07.750169 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.750137    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-home\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.750307 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.750190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-tmp-dir\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.750307 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.750215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-kserve-provision-location\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.750307 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.750234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc79eb50-1521-4f4a-9028-4c2f896cea29-tls-certs\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.750307 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.750268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-dshm\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.750307 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.750291    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n2cx\" (UniqueName: \"kubernetes.io/projected/cc79eb50-1521-4f4a-9028-4c2f896cea29-kube-api-access-9n2cx\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.750494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.750327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-model-cache\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.851477 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.851441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-kserve-provision-location\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.851477 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.851480    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc79eb50-1521-4f4a-9028-4c2f896cea29-tls-certs\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.851737 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.851511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-dshm\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.851737 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.851621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n2cx\" (UniqueName: \"kubernetes.io/projected/cc79eb50-1521-4f4a-9028-4c2f896cea29-kube-api-access-9n2cx\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.851737 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.851683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-model-cache\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.851737 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.851724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-home\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.851974 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.851771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-tmp-dir\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.851974 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.851914    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-kserve-provision-location\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.852127 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.852106    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-tmp-dir\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.852191 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.852140    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-model-cache\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.852241 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.852183    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-home\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.853873 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.853844    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-dshm\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.854063 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.854047    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc79eb50-1521-4f4a-9028-4c2f896cea29-tls-certs\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:07.860653 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:07.860626    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n2cx\" (UniqueName: \"kubernetes.io/projected/cc79eb50-1521-4f4a-9028-4c2f896cea29-kube-api-access-9n2cx\") pod \"router-with-refs-test-kserve-6b9f4b8d6b-97gl5\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:08.001563 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:08.001529    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:08.147057 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:08.147022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"]
Jul 09 10:09:08.151293 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:09:08.151254    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc79eb50_1521_4f4a_9028_4c2f896cea29.slice/crio-a905196f3aebf416d6f36f90308bb5d90099f2acbc2d7e3ec4020b4ac5ba9cbe WatchSource:0}: Error finding container a905196f3aebf416d6f36f90308bb5d90099f2acbc2d7e3ec4020b4ac5ba9cbe: Status 404 returned error can't find the container with id a905196f3aebf416d6f36f90308bb5d90099f2acbc2d7e3ec4020b4ac5ba9cbe
Jul 09 10:09:08.660656 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:08.660617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" event={"ID":"cc79eb50-1521-4f4a-9028-4c2f896cea29","Type":"ContainerStarted","Data":"19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185"}
Jul 09 10:09:08.660866 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:08.660699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" event={"ID":"cc79eb50-1521-4f4a-9028-4c2f896cea29","Type":"ContainerStarted","Data":"a905196f3aebf416d6f36f90308bb5d90099f2acbc2d7e3ec4020b4ac5ba9cbe"}
Jul 09 10:09:12.686820 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:12.686787    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerID="19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185" exitCode=0
Jul 09 10:09:12.687250 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:12.686866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" event={"ID":"cc79eb50-1521-4f4a-9028-4c2f896cea29","Type":"ContainerDied","Data":"19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185"}
Jul 09 10:09:13.693045 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:13.693004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" event={"ID":"cc79eb50-1521-4f4a-9028-4c2f896cea29","Type":"ContainerStarted","Data":"d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405"}
Jul 09 10:09:13.718038 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:13.717975    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podStartSLOduration=2.97545736 podStartE2EDuration="6.717959731s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.742502371s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:09:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:09:13.715607793 +0000 UTC m=+2146.040570092" watchObservedRunningTime="2026-07-09 10:09:13.717959731 +0000 UTC m=+2146.042922014"
Jul 09 10:09:14.002120 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:14.002077    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:17.085384 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:17.085339    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:09:18.002634 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:18.002601    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:09:18.004369 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:18.004344    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:09:27.085308 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:27.085218    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:09:28.002648 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:28.002608    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:09:37.085277 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:37.085230    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:09:38.002614 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:38.002573    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:09:47.085226 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:47.085181    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:09:48.002249 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:48.002203    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:09:57.084904 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:57.084846    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:09:58.002923 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:09:58.002865    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:10:07.095118 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:10:07.095087    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:10:07.106489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:10:07.106460    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:10:08.002373 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:10:08.002338    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:10:18.002624 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:10:18.002586    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:10:28.002757 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:10:28.002701    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:10:38.002171 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:10:38.002131    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:10:48.002821 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:10:48.002779    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:10:58.002757 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:10:58.002665    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:11:08.011579 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:11:08.011551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:11:08.019203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:11:08.019177    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:13:16.942735 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:16.942648    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"]
Jul 09 10:13:30.110398 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:30.110373    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:13:30.114835 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:30.114815    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:13:31.966588 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:31.966543    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" containerID="cri-o://d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405" gracePeriod=45
Jul 09 10:13:32.012180 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.012141    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": dial tcp 10.133.0.44:8000: connect: connection refused"
Jul 09 10:13:32.733416 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.733394    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:13:32.766733 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.766671    2581 generic.go:356] "Generic (PLEG): container finished" podID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerID="d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405" exitCode=0
Jul 09 10:13:32.766733 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.766714    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" event={"ID":"cc79eb50-1521-4f4a-9028-4c2f896cea29","Type":"ContainerDied","Data":"d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405"}
Jul 09 10:13:32.766733 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.766719    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"
Jul 09 10:13:32.766946 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.766760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5" event={"ID":"cc79eb50-1521-4f4a-9028-4c2f896cea29","Type":"ContainerDied","Data":"a905196f3aebf416d6f36f90308bb5d90099f2acbc2d7e3ec4020b4ac5ba9cbe"}
Jul 09 10:13:32.766946 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.766785    2581 scope.go:117] "RemoveContainer" containerID="d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405"
Jul 09 10:13:32.774704 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.774682    2581 scope.go:117] "RemoveContainer" containerID="19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185"
Jul 09 10:13:32.780972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.780941    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc79eb50-1521-4f4a-9028-4c2f896cea29-tls-certs\") pod \"cc79eb50-1521-4f4a-9028-4c2f896cea29\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") "
Jul 09 10:13:32.781082 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.780977    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-dshm\") pod \"cc79eb50-1521-4f4a-9028-4c2f896cea29\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") "
Jul 09 10:13:32.781082 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.780998    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-tmp-dir\") pod \"cc79eb50-1521-4f4a-9028-4c2f896cea29\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") "
Jul 09 10:13:32.781082 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.781021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-kserve-provision-location\") pod \"cc79eb50-1521-4f4a-9028-4c2f896cea29\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") "
Jul 09 10:13:32.781082 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.781063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n2cx\" (UniqueName: \"kubernetes.io/projected/cc79eb50-1521-4f4a-9028-4c2f896cea29-kube-api-access-9n2cx\") pod \"cc79eb50-1521-4f4a-9028-4c2f896cea29\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") "
Jul 09 10:13:32.781293 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.781086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-model-cache\") pod \"cc79eb50-1521-4f4a-9028-4c2f896cea29\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") "
Jul 09 10:13:32.781293 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.781130    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-home\") pod \"cc79eb50-1521-4f4a-9028-4c2f896cea29\" (UID: \"cc79eb50-1521-4f4a-9028-4c2f896cea29\") "
Jul 09 10:13:32.781904 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.781847    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc79eb50-1521-4f4a-9028-4c2f896cea29" (UID: "cc79eb50-1521-4f4a-9028-4c2f896cea29"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:13:32.783283 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.782302    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:13:32.783434 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.783399    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-home" (OuterVolumeSpecName: "home") pod "cc79eb50-1521-4f4a-9028-4c2f896cea29" (UID: "cc79eb50-1521-4f4a-9028-4c2f896cea29"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:13:32.784039 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.783524    2581 scope.go:117] "RemoveContainer" containerID="d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405"
Jul 09 10:13:32.785624 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.784135    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc79eb50-1521-4f4a-9028-4c2f896cea29-kube-api-access-9n2cx" (OuterVolumeSpecName: "kube-api-access-9n2cx") pod "cc79eb50-1521-4f4a-9028-4c2f896cea29" (UID: "cc79eb50-1521-4f4a-9028-4c2f896cea29"). InnerVolumeSpecName "kube-api-access-9n2cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:13:32.790822 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.790775    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-dshm" (OuterVolumeSpecName: "dshm") pod "cc79eb50-1521-4f4a-9028-4c2f896cea29" (UID: "cc79eb50-1521-4f4a-9028-4c2f896cea29"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:13:32.790989 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:13:32.790962    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405\": container with ID starting with d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405 not found: ID does not exist" containerID="d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405"
Jul 09 10:13:32.791066 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.790991    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405"} err="failed to get container status \"d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405\": rpc error: code = NotFound desc = could not find container \"d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405\": container with ID starting with d1ccd735f422e6dcd6cdc813f1f7f1d4c614fbc604ea6cfdcc32b472fa56f405 not found: ID does not exist"
Jul 09 10:13:32.791066 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.791017    2581 scope.go:117] "RemoveContainer" containerID="19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185"
Jul 09 10:13:32.791527 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:13:32.791504    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185\": container with ID starting with 19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185 not found: ID does not exist" containerID="19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185"
Jul 09 10:13:32.791609 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.791533    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185"} err="failed to get container status \"19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185\": rpc error: code = NotFound desc = could not find container \"19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185\": container with ID starting with 19e944ff3a76d5679303fe8602948cce1d5e442c9d8773e70ac21f3cd1cbc185 not found: ID does not exist"
Jul 09 10:13:32.792412 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.792390    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc79eb50-1521-4f4a-9028-4c2f896cea29-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc79eb50-1521-4f4a-9028-4c2f896cea29" (UID: "cc79eb50-1521-4f4a-9028-4c2f896cea29"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:13:32.801189 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.801168    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc79eb50-1521-4f4a-9028-4c2f896cea29" (UID: "cc79eb50-1521-4f4a-9028-4c2f896cea29"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:13:32.856231 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.856200    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc79eb50-1521-4f4a-9028-4c2f896cea29" (UID: "cc79eb50-1521-4f4a-9028-4c2f896cea29"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:13:32.882995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.882974    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc79eb50-1521-4f4a-9028-4c2f896cea29-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:13:32.882995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.882994    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:13:32.883116 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.883002    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:13:32.883116 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.883011    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:13:32.883116 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.883020    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n2cx\" (UniqueName: \"kubernetes.io/projected/cc79eb50-1521-4f4a-9028-4c2f896cea29-kube-api-access-9n2cx\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:13:32.883116 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:32.883029    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc79eb50-1521-4f4a-9028-4c2f896cea29-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:13:33.089792 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:33.089756    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"]
Jul 09 10:13:33.095480 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:33.095456    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-6b9f4b8d6b-97gl5"]
Jul 09 10:13:34.228091 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:34.228061    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" path="/var/lib/kubelet/pods/cc79eb50-1521-4f4a-9028-4c2f896cea29/volumes"
Jul 09 10:13:43.151596 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.151560    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"]
Jul 09 10:13:43.151984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.151835    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="storage-initializer"
Jul 09 10:13:43.151984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.151846    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="storage-initializer"
Jul 09 10:13:43.151984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.151858    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main"
Jul 09 10:13:43.151984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.151864    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main"
Jul 09 10:13:43.151984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.151929    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc79eb50-1521-4f4a-9028-4c2f896cea29" containerName="main"
Jul 09 10:13:43.155106 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.155081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.159153 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.159130    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 09 10:13:43.159543 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.159520    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 09 10:13:43.159807 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.159787    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-jh55j\""
Jul 09 10:13:43.170341 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.170318    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"]
Jul 09 10:13:43.173700 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.173676    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"]
Jul 09 10:13:43.176672 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.176642    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.176777 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.176688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.176777 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.176752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.176908 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.176789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.176908 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.176817    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0904148-f213-45f5-b3df-16850023d4a6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.176908 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.176848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.177076 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.176882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj2sh\" (UniqueName: \"kubernetes.io/projected/d0904148-f213-45f5-b3df-16850023d4a6-kube-api-access-jj2sh\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.177281 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.177265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.195558 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.195532    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"]
Jul 09 10:13:43.278204 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vhw9\" (UniqueName: \"kubernetes.io/projected/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kube-api-access-6vhw9\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.278204 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.278394 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278229    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.278394 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.278394 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.278394 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278388    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.278576 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278423    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.278576 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.278576 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.278576 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.278576 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278524    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0904148-f213-45f5-b3df-16850023d4a6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.278576 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.278869 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278582    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.278869 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj2sh\" (UniqueName: \"kubernetes.io/projected/d0904148-f213-45f5-b3df-16850023d4a6-kube-api-access-jj2sh\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.278869 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.278737    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.279080 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.279058    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.279080 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.279071    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.279202 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.279111    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.280690 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.280665    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.281026 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.281011    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0904148-f213-45f5-b3df-16850023d4a6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.286230 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.286205    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj2sh\" (UniqueName: \"kubernetes.io/projected/d0904148-f213-45f5-b3df-16850023d4a6-kube-api-access-jj2sh\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.379275 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.379470 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.379470 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.379470 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379325    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.379470 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379374    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vhw9\" (UniqueName: \"kubernetes.io/projected/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kube-api-access-6vhw9\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.379470 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379404    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.379470 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.379810 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379770    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.379810 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379792    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.379935 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379871    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.379935 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.379877    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.381473 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.381456    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.381777 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.381759    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.387451 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.387427    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vhw9\" (UniqueName: \"kubernetes.io/projected/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kube-api-access-6vhw9\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.467334 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.467298    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.486865 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.486827    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:43.619483 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.619458    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"]
Jul 09 10:13:43.621601 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:13:43.621568    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0904148_f213_45f5_b3df_16850023d4a6.slice/crio-517f578b745f5527888d6c44c1971040a1f1423b792002fb6ec74f393770ddc0 WatchSource:0}: Error finding container 517f578b745f5527888d6c44c1971040a1f1423b792002fb6ec74f393770ddc0: Status 404 returned error can't find the container with id 517f578b745f5527888d6c44c1971040a1f1423b792002fb6ec74f393770ddc0
Jul 09 10:13:43.641864 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.641843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"]
Jul 09 10:13:43.658977 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:13:43.658949    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5c820c_df5b_4f8a_8d72_dc8f1db2c064.slice/crio-45b9254b017ccc9bc5e5c791d0247b4cf3c7e107a8f205b0426c382718d46c87 WatchSource:0}: Error finding container 45b9254b017ccc9bc5e5c791d0247b4cf3c7e107a8f205b0426c382718d46c87: Status 404 returned error can't find the container with id 45b9254b017ccc9bc5e5c791d0247b4cf3c7e107a8f205b0426c382718d46c87
Jul 09 10:13:43.812241 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.812145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" event={"ID":"d0904148-f213-45f5-b3df-16850023d4a6","Type":"ContainerStarted","Data":"f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11"}
Jul 09 10:13:43.812241 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.812209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" event={"ID":"d0904148-f213-45f5-b3df-16850023d4a6","Type":"ContainerStarted","Data":"517f578b745f5527888d6c44c1971040a1f1423b792002fb6ec74f393770ddc0"}
Jul 09 10:13:43.812486 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.812350    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:43.813613 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.813583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" event={"ID":"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064","Type":"ContainerStarted","Data":"04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53"}
Jul 09 10:13:43.813709 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:43.813618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" event={"ID":"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064","Type":"ContainerStarted","Data":"45b9254b017ccc9bc5e5c791d0247b4cf3c7e107a8f205b0426c382718d46c87"}
Jul 09 10:13:44.825665 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:44.825455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" event={"ID":"d0904148-f213-45f5-b3df-16850023d4a6","Type":"ContainerStarted","Data":"47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7"}
Jul 09 10:13:47.841299 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:47.841262    2581 generic.go:356] "Generic (PLEG): container finished" podID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerID="04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53" exitCode=0
Jul 09 10:13:47.841728 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:47.841335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" event={"ID":"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064","Type":"ContainerDied","Data":"04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53"}
Jul 09 10:13:47.843337 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:47.843315    2581 generic.go:356] "Generic (PLEG): container finished" podID="d0904148-f213-45f5-b3df-16850023d4a6" containerID="47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7" exitCode=0
Jul 09 10:13:47.843437 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:47.843374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" event={"ID":"d0904148-f213-45f5-b3df-16850023d4a6","Type":"ContainerDied","Data":"47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7"}
Jul 09 10:13:48.850234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:48.850199    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" event={"ID":"d0904148-f213-45f5-b3df-16850023d4a6","Type":"ContainerStarted","Data":"dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53"}
Jul 09 10:13:48.852310 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:48.852280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" event={"ID":"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064","Type":"ContainerStarted","Data":"e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235"}
Jul 09 10:13:48.873821 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:48.873771    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podStartSLOduration=2.752815128 podStartE2EDuration="5.87375372s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.120938592s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:13:48.873035131 +0000 UTC m=+2421.197997416" watchObservedRunningTime="2026-07-09 10:13:48.87375372 +0000 UTC m=+2421.198716009"
Jul 09 10:13:48.892665 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:48.892604    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podStartSLOduration=2.53405801 podStartE2EDuration="5.892584706s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.358526696s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:13:48.8917802 +0000 UTC m=+2421.216742497" watchObservedRunningTime="2026-07-09 10:13:48.892584706 +0000 UTC m=+2421.217546991"
Jul 09 10:13:53.468300 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:53.468209    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:53.468873 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:53.468854    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:53.469453 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:53.469416    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:13:53.487060 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:53.487031    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:53.487289 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:53.487275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:13:53.488444 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:53.488406    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:13:53.491608 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:53.491590    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:13:59.633559 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:59.633516    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 10:13:59.657690 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:59.657653    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"]
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:13:59.673752    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]:  > containerID="e1e775f7ab465e13e5ff0a79ec350b6a7132ad91cdf9fe24e0e22fcf9e2ee643" cmd=["/bin/sleep","15"]
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:59.673796    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj" message="" err=<
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]:  >
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:13:59.673808    2581 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj" podUID="8307803a-aaf8-403d-bcb7-07fe84c92555" containerName="main" containerID="cri-o://e1e775f7ab465e13e5ff0a79ec350b6a7132ad91cdf9fe24e0e22fcf9e2ee643"
Jul 09 10:13:59.674103 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:59.673835    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj" podUID="8307803a-aaf8-403d-bcb7-07fe84c92555" containerName="main" containerID="cri-o://e1e775f7ab465e13e5ff0a79ec350b6a7132ad91cdf9fe24e0e22fcf9e2ee643" gracePeriod=60
Jul 09 10:13:59.907013 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:59.906926    2581 generic.go:356] "Generic (PLEG): container finished" podID="8307803a-aaf8-403d-bcb7-07fe84c92555" containerID="e1e775f7ab465e13e5ff0a79ec350b6a7132ad91cdf9fe24e0e22fcf9e2ee643" exitCode=0
Jul 09 10:13:59.907013 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:59.906980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj" event={"ID":"8307803a-aaf8-403d-bcb7-07fe84c92555","Type":"ContainerDied","Data":"e1e775f7ab465e13e5ff0a79ec350b6a7132ad91cdf9fe24e0e22fcf9e2ee643"}
Jul 09 10:13:59.967330 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:13:59.967296    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:14:00.037948 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.037486    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8307803a-aaf8-403d-bcb7-07fe84c92555-tls-certs\") pod \"8307803a-aaf8-403d-bcb7-07fe84c92555\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") "
Jul 09 10:14:00.037948 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.037558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9xmb\" (UniqueName: \"kubernetes.io/projected/8307803a-aaf8-403d-bcb7-07fe84c92555-kube-api-access-k9xmb\") pod \"8307803a-aaf8-403d-bcb7-07fe84c92555\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") "
Jul 09 10:14:00.037948 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.037658    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8307803a-aaf8-403d-bcb7-07fe84c92555-tokenizer-uds\") pod \"8307803a-aaf8-403d-bcb7-07fe84c92555\" (UID: \"8307803a-aaf8-403d-bcb7-07fe84c92555\") "
Jul 09 10:14:00.038297 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.038103    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8307803a-aaf8-403d-bcb7-07fe84c92555-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8307803a-aaf8-403d-bcb7-07fe84c92555" (UID: "8307803a-aaf8-403d-bcb7-07fe84c92555"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:14:00.039754 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.039728    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8307803a-aaf8-403d-bcb7-07fe84c92555-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8307803a-aaf8-403d-bcb7-07fe84c92555" (UID: "8307803a-aaf8-403d-bcb7-07fe84c92555"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:14:00.040214 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.040196    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8307803a-aaf8-403d-bcb7-07fe84c92555-kube-api-access-k9xmb" (OuterVolumeSpecName: "kube-api-access-k9xmb") pod "8307803a-aaf8-403d-bcb7-07fe84c92555" (UID: "8307803a-aaf8-403d-bcb7-07fe84c92555"). InnerVolumeSpecName "kube-api-access-k9xmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:14:00.138608 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.138570    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8307803a-aaf8-403d-bcb7-07fe84c92555-tokenizer-uds\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:00.138608 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.138602    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8307803a-aaf8-403d-bcb7-07fe84c92555-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:00.138608 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.138612    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9xmb\" (UniqueName: \"kubernetes.io/projected/8307803a-aaf8-403d-bcb7-07fe84c92555-kube-api-access-k9xmb\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:00.913266 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.913232    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"
Jul 09 10:14:00.913747 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.913264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj" event={"ID":"8307803a-aaf8-403d-bcb7-07fe84c92555","Type":"ContainerDied","Data":"946be4d600dfe7c4a0a09cbf56f40145052827b3df847f09c82b8749561443a1"}
Jul 09 10:14:00.913747 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.913338    2581 scope.go:117] "RemoveContainer" containerID="e1e775f7ab465e13e5ff0a79ec350b6a7132ad91cdf9fe24e0e22fcf9e2ee643"
Jul 09 10:14:00.930210 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.930183    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"]
Jul 09 10:14:00.939229 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:00.939203    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc1dc4269d1ada5f2d28562215d180c57f-kserve-router-schebp4pj"]
Jul 09 10:14:02.230395 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:02.230361    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8307803a-aaf8-403d-bcb7-07fe84c92555" path="/var/lib/kubelet/pods/8307803a-aaf8-403d-bcb7-07fe84c92555/volumes"
Jul 09 10:14:03.469100 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:03.469051    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:14:03.487237 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:03.487200    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:14:13.468016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:13.467956    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:14:13.488128 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:13.488092    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:14:14.681448 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:14.681381    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" containerID="cri-o://29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6" gracePeriod=45
Jul 09 10:14:15.096014 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.095958    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": dial tcp 10.133.0.41:8000: connect: connection refused"
Jul 09 10:14:15.460494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.460466    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:14:15.593409 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.593369    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-home\") pod \"70322401-e103-4e95-8134-0c36df44de20\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") "
Jul 09 10:14:15.593613 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.593438    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70322401-e103-4e95-8134-0c36df44de20-tls-certs\") pod \"70322401-e103-4e95-8134-0c36df44de20\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") "
Jul 09 10:14:15.593613 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.593457    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9b97\" (UniqueName: \"kubernetes.io/projected/70322401-e103-4e95-8134-0c36df44de20-kube-api-access-n9b97\") pod \"70322401-e103-4e95-8134-0c36df44de20\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") "
Jul 09 10:14:15.593613 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.593555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"70322401-e103-4e95-8134-0c36df44de20\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") "
Jul 09 10:14:15.593613 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.593605    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-model-cache\") pod \"70322401-e103-4e95-8134-0c36df44de20\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") "
Jul 09 10:14:15.593842 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.593643    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-dshm\") pod \"70322401-e103-4e95-8134-0c36df44de20\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") "
Jul 09 10:14:15.593842 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.593700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-tmp-dir\") pod \"70322401-e103-4e95-8134-0c36df44de20\" (UID: \"70322401-e103-4e95-8134-0c36df44de20\") "
Jul 09 10:14:15.594038 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.593993    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-home" (OuterVolumeSpecName: "home") pod "70322401-e103-4e95-8134-0c36df44de20" (UID: "70322401-e103-4e95-8134-0c36df44de20"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:14:15.594038 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.594010    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-model-cache" (OuterVolumeSpecName: "model-cache") pod "70322401-e103-4e95-8134-0c36df44de20" (UID: "70322401-e103-4e95-8134-0c36df44de20"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:14:15.596573 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.596543    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70322401-e103-4e95-8134-0c36df44de20-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70322401-e103-4e95-8134-0c36df44de20" (UID: "70322401-e103-4e95-8134-0c36df44de20"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:14:15.596573 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.596543    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-dshm" (OuterVolumeSpecName: "dshm") pod "70322401-e103-4e95-8134-0c36df44de20" (UID: "70322401-e103-4e95-8134-0c36df44de20"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:14:15.596573 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.596562    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70322401-e103-4e95-8134-0c36df44de20-kube-api-access-n9b97" (OuterVolumeSpecName: "kube-api-access-n9b97") pod "70322401-e103-4e95-8134-0c36df44de20" (UID: "70322401-e103-4e95-8134-0c36df44de20"). InnerVolumeSpecName "kube-api-access-n9b97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:14:15.597129 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.597104    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522" (OuterVolumeSpecName: "kserve-pvc-source") pod "70322401-e103-4e95-8134-0c36df44de20" (UID: "70322401-e103-4e95-8134-0c36df44de20"). InnerVolumeSpecName "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 10:14:15.607050 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.607014    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70322401-e103-4e95-8134-0c36df44de20" (UID: "70322401-e103-4e95-8134-0c36df44de20"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:14:15.694531 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.694493    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:15.694531 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.694523    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:15.694531 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.694532    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:15.694531 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.694540    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70322401-e103-4e95-8134-0c36df44de20-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:15.695034 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.694547    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70322401-e103-4e95-8134-0c36df44de20-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:15.695034 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.694557    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9b97\" (UniqueName: \"kubernetes.io/projected/70322401-e103-4e95-8134-0c36df44de20-kube-api-access-n9b97\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:15.982458 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.982429    2581 generic.go:356] "Generic (PLEG): container finished" podID="70322401-e103-4e95-8134-0c36df44de20" containerID="29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6" exitCode=0
Jul 09 10:14:15.982638 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.982485    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"70322401-e103-4e95-8134-0c36df44de20","Type":"ContainerDied","Data":"29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6"}
Jul 09 10:14:15.982638 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.982497    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 09 10:14:15.982638 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.982559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"70322401-e103-4e95-8134-0c36df44de20","Type":"ContainerDied","Data":"612fb59b8ef32bacf3db79c1ea4cc9138a11fa1a442dda0fc34c6e1ba77df405"}
Jul 09 10:14:15.982638 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.982587    2581 scope.go:117] "RemoveContainer" containerID="29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6"
Jul 09 10:14:15.991060 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.991038    2581 scope.go:117] "RemoveContainer" containerID="29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6"
Jul 09 10:14:15.991366 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:14:15.991349    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6\": container with ID starting with 29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6 not found: ID does not exist" containerID="29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6"
Jul 09 10:14:15.991464 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:15.991373    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6"} err="failed to get container status \"29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6\": rpc error: code = NotFound desc = could not find container \"29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6\": container with ID starting with 29ed877a3c6f6c85227a41e271e0fcca673eeacd46ff49abab89f5c21e5b71e6 not found: ID does not exist"
Jul 09 10:14:16.007533 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:16.007506    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 10:14:16.012757 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:16.012732    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 09 10:14:16.029177 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:16.029150    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 10:14:16.229032 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:16.228994    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70322401-e103-4e95-8134-0c36df44de20" path="/var/lib/kubelet/pods/70322401-e103-4e95-8134-0c36df44de20/volumes"
Jul 09 10:14:23.468733 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:23.468688    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:14:23.487368 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:23.487333    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:14:31.052920 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.052853    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" containerName="main" containerID="cri-o://40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac" gracePeriod=45
Jul 09 10:14:31.731059 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.731032    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:14:31.836618 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.836522    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-dshm\") pod \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") "
Jul 09 10:14:31.836618 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.836577    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-model-cache\") pod \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") "
Jul 09 10:14:31.836849 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.836635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmrj6\" (UniqueName: \"kubernetes.io/projected/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-kube-api-access-tmrj6\") pod \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") "
Jul 09 10:14:31.836849 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.836681    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-home\") pod \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") "
Jul 09 10:14:31.836849 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.836712    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tmp-dir\") pod \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") "
Jul 09 10:14:31.836849 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.836753    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tls-certs\") pod \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") "
Jul 09 10:14:31.837112 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.836852    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") pod \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\" (UID: \"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21\") "
Jul 09 10:14:31.837112 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.836869    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-model-cache" (OuterVolumeSpecName: "model-cache") pod "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" (UID: "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:14:31.837112 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.837091    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:31.837412 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.837352    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-home" (OuterVolumeSpecName: "home") pod "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" (UID: "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:14:31.839436 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.839397    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-dshm" (OuterVolumeSpecName: "dshm") pod "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" (UID: "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:14:31.839566 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.839475    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" (UID: "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:14:31.839796 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.839772    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-kube-api-access-tmrj6" (OuterVolumeSpecName: "kube-api-access-tmrj6") pod "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" (UID: "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21"). InnerVolumeSpecName "kube-api-access-tmrj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:14:31.840481 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.840458    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522" (OuterVolumeSpecName: "kserve-pvc-source") pod "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" (UID: "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21"). InnerVolumeSpecName "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 09 10:14:31.849322 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.849284    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" (UID: "dfb12740-c438-4ed8-b9a1-d11aa0a2ba21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:14:31.938511 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.938468    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmrj6\" (UniqueName: \"kubernetes.io/projected/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-kube-api-access-tmrj6\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:31.938511 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.938508    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:31.938708 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.938526    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:31.938708 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.938540    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:31.938708 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.938581    2581 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") on node \"ip-10-0-129-157.ec2.internal\" "
Jul 09 10:14:31.938708 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.938595    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:14:31.966979 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:31.966955    2581 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522") on node "ip-10-0-129-157.ec2.internal"
Jul 09 10:14:32.039502 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.039469    2581 reconciler_common.go:299] "Volume detached for volume \"pvc-63da8511-2b18-4fae-9ef5-e3e1723a6f9b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bc464e1a50b69522\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"csi-a4decca2aaec13f4dfab50e9313a126be0c884fb9dbeffffba32cc7747037064\""
Jul 09 10:14:32.064381 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.064346    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" containerID="40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac" exitCode=0
Jul 09 10:14:32.064824 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.064387    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 09 10:14:32.064824 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.064436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21","Type":"ContainerDied","Data":"40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac"}
Jul 09 10:14:32.064824 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.064495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"dfb12740-c438-4ed8-b9a1-d11aa0a2ba21","Type":"ContainerDied","Data":"16ce2a070d8d8a3f59382c51e3f64d161ba41101d95cb660a225475b5cf84b33"}
Jul 09 10:14:32.064824 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.064530    2581 scope.go:117] "RemoveContainer" containerID="40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac"
Jul 09 10:14:32.074872 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.074856    2581 scope.go:117] "RemoveContainer" containerID="40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac"
Jul 09 10:14:32.075174 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:14:32.075156    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac\": container with ID starting with 40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac not found: ID does not exist" containerID="40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac"
Jul 09 10:14:32.075238 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.075184    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac"} err="failed to get container status \"40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac\": rpc error: code = NotFound desc = could not find container \"40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac\": container with ID starting with 40a545653d892e1eda3b4ea08cdf37c60bfb7c73476cdc7fb63044d7769dceac not found: ID does not exist"
Jul 09 10:14:32.085716 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.085697    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 10:14:32.089376 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.089329    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 09 10:14:32.230044 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:32.230016    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" path="/var/lib/kubelet/pods/dfb12740-c438-4ed8-b9a1-d11aa0a2ba21/volumes"
Jul 09 10:14:33.468473 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:33.468429    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:14:33.488085 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:33.488048    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:14:43.468547 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:43.468493    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:14:43.487616 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:43.487583    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:14:45.529395 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.529356    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"]
Jul 09 10:14:45.529851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.529637    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8307803a-aaf8-403d-bcb7-07fe84c92555" containerName="main"
Jul 09 10:14:45.529851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.529647    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8307803a-aaf8-403d-bcb7-07fe84c92555" containerName="main"
Jul 09 10:14:45.529851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.529656    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main"
Jul 09 10:14:45.529851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.529662    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main"
Jul 09 10:14:45.529851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.529674    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" containerName="main"
Jul 09 10:14:45.529851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.529679    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" containerName="main"
Jul 09 10:14:45.529851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.529726    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8307803a-aaf8-403d-bcb7-07fe84c92555" containerName="main"
Jul 09 10:14:45.529851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.529734    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfb12740-c438-4ed8-b9a1-d11aa0a2ba21" containerName="main"
Jul 09 10:14:45.529851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.529742    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="70322401-e103-4e95-8134-0c36df44de20" containerName="main"
Jul 09 10:14:45.531821 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.531805    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.535148 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.535130    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 09 10:14:45.551532 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.551502    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"]
Jul 09 10:14:45.656575 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.656535    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kserve-provision-location\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.656773 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.656601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tmp-dir\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.656773 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.656646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-dshm\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.656773 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.656675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-home\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.656773 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.656719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-model-cache\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.656773 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.656747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tls-certs\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.656994 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.656776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmb5q\" (UniqueName: \"kubernetes.io/projected/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kube-api-access-xmb5q\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757260 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tmp-dir\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757436 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757279    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-dshm\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757436 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-home\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757436 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-model-cache\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757436 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tls-certs\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757436 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmb5q\" (UniqueName: \"kubernetes.io/projected/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kube-api-access-xmb5q\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757695 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kserve-provision-location\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757753 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757734    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-model-cache\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757807 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757748    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-home\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757859 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757809    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kserve-provision-location\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.757956 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.757937    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tmp-dir\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.759834 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.759805    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-dshm\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.760179 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.760156    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tls-certs\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.774562 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.774526    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmb5q\" (UniqueName: \"kubernetes.io/projected/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kube-api-access-xmb5q\") pod \"stop-feature-test-kserve-5446b4db4b-x7nl4\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:45.845274 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:45.845185    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:46.002943 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:46.002913    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"]
Jul 09 10:14:46.005134 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:14:46.005108    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be8d58c_0e57_463d_8e0d_1bb3661481bd.slice/crio-42a95f152811c385707cd44bca23f302a991c29131b69acc6d675463c51e2c77 WatchSource:0}: Error finding container 42a95f152811c385707cd44bca23f302a991c29131b69acc6d675463c51e2c77: Status 404 returned error can't find the container with id 42a95f152811c385707cd44bca23f302a991c29131b69acc6d675463c51e2c77
Jul 09 10:14:46.133868 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:46.133773    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" event={"ID":"0be8d58c-0e57-463d-8e0d-1bb3661481bd","Type":"ContainerStarted","Data":"d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9"}
Jul 09 10:14:46.133868 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:46.133829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" event={"ID":"0be8d58c-0e57-463d-8e0d-1bb3661481bd","Type":"ContainerStarted","Data":"42a95f152811c385707cd44bca23f302a991c29131b69acc6d675463c51e2c77"}
Jul 09 10:14:50.165058 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:50.165019    2581 generic.go:356] "Generic (PLEG): container finished" podID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerID="d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9" exitCode=0
Jul 09 10:14:50.165528 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:50.165067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" event={"ID":"0be8d58c-0e57-463d-8e0d-1bb3661481bd","Type":"ContainerDied","Data":"d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9"}
Jul 09 10:14:51.172612 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:51.172576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" event={"ID":"0be8d58c-0e57-463d-8e0d-1bb3661481bd","Type":"ContainerStarted","Data":"8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5"}
Jul 09 10:14:51.196603 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:51.196538    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podStartSLOduration=2.324226569 podStartE2EDuration="6.196517995s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.872291426s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:14:51.194687833 +0000 UTC m=+2483.519650122" watchObservedRunningTime="2026-07-09 10:14:51.196517995 +0000 UTC m=+2483.521480281"
Jul 09 10:14:51.856541 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:51.856499    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:53.468578 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:53.468517    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:14:53.488169 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:53.488132    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:14:55.846046 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:55.846016    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:14:55.847883 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:14:55.847850    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:15:03.468741 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:03.468694    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:15:03.487317 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:03.487283    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:15:05.846379 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:05.846333    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:15:13.468457 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:13.468414    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:15:13.488185 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:13.488143    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:15:15.846310 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:15.846272    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:15:23.468287 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:23.468192    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:15:23.487910 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:23.487851    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:15:25.846316 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:25.846260    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:15:33.468199 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:33.468143    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:15:33.488085 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:33.488038    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:15:35.845984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:35.845935    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:15:43.468268 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:43.468222    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:15:43.487257 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:43.487214    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:15:45.846229 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:45.846192    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:15:53.467958 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:53.467915    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:15:53.488157 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:53.488123    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:15:55.846533 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:15:55.846485    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:16:03.468463 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:03.468417    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:16:03.487953 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:03.487913    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:16:05.846393 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:05.846339    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:16:13.468609 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:13.468555    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:16:13.488280 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:13.488240    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:16:15.846292 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:15.846236    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:16:23.468560 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:23.468508    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8001/health\": dial tcp 10.133.0.45:8001: connect: connection refused"
Jul 09 10:16:23.488052 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:23.488004    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 09 10:16:25.846507 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:25.846460    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:16:33.478575 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:33.478540    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:16:33.491130 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:33.491105    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:16:33.503346 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:33.503315    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:16:33.512366 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:33.512338    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:16:35.846475 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:35.846438    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:16:45.846030 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:45.845989    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:16:48.942078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:48.942042    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"]
Jul 09 10:16:48.947494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:48.947465    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"]
Jul 09 10:16:55.846201 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:16:55.846105    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:17:03.965858 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:03.965812    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main" containerID="cri-o://dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53" gracePeriod=45
Jul 09 10:17:03.972864 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:03.972832    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main" containerID="cri-o://e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235" gracePeriod=45
Jul 09 10:17:05.077097 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.076978    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="llm-d-routing-sidecar" containerID="cri-o://f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11" gracePeriod=44
Jul 09 10:17:05.246018 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.245950    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:17:05.351267 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.351232    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:17:05.367799 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.367776    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kserve-provision-location\") pod \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") "
Jul 09 10:17:05.367955 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.367808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-home\") pod \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") "
Jul 09 10:17:05.367955 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.367846    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tls-certs\") pod \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") "
Jul 09 10:17:05.367955 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.367897    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vhw9\" (UniqueName: \"kubernetes.io/projected/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kube-api-access-6vhw9\") pod \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") "
Jul 09 10:17:05.367955 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.367937    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-model-cache\") pod \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") "
Jul 09 10:17:05.368186 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.367961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-dshm\") pod \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") "
Jul 09 10:17:05.368186 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.368048    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tmp-dir\") pod \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\" (UID: \"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064\") "
Jul 09 10:17:05.368298 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.368234    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" (UID: "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:05.368455 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.368418    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-home" (OuterVolumeSpecName: "home") pod "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" (UID: "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:05.368716 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.368696    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.368789 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.368723    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.370556 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.370519    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-dshm" (OuterVolumeSpecName: "dshm") pod "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" (UID: "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:05.370658 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.370572    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" (UID: "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:17:05.370658 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.370585    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kube-api-access-6vhw9" (OuterVolumeSpecName: "kube-api-access-6vhw9") pod "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" (UID: "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064"). InnerVolumeSpecName "kube-api-access-6vhw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:17:05.382294 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.382262    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" (UID: "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:05.446451 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.446387    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" (UID: "3c5c820c-df5b-4f8a-8d72-dc8f1db2c064"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:05.469532 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.469511    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-tmp-dir\") pod \"d0904148-f213-45f5-b3df-16850023d4a6\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") "
Jul 09 10:17:05.469643 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.469560    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-kserve-provision-location\") pod \"d0904148-f213-45f5-b3df-16850023d4a6\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") "
Jul 09 10:17:05.469643 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.469588    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj2sh\" (UniqueName: \"kubernetes.io/projected/d0904148-f213-45f5-b3df-16850023d4a6-kube-api-access-jj2sh\") pod \"d0904148-f213-45f5-b3df-16850023d4a6\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") "
Jul 09 10:17:05.469643 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.469621    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-dshm\") pod \"d0904148-f213-45f5-b3df-16850023d4a6\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") "
Jul 09 10:17:05.469750 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.469682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-model-cache\") pod \"d0904148-f213-45f5-b3df-16850023d4a6\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") "
Jul 09 10:17:05.470092 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.469799    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0904148-f213-45f5-b3df-16850023d4a6-tls-certs\") pod \"d0904148-f213-45f5-b3df-16850023d4a6\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") "
Jul 09 10:17:05.470092 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.469907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-home\") pod \"d0904148-f213-45f5-b3df-16850023d4a6\" (UID: \"d0904148-f213-45f5-b3df-16850023d4a6\") "
Jul 09 10:17:05.470092 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.469923    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "d0904148-f213-45f5-b3df-16850023d4a6" (UID: "d0904148-f213-45f5-b3df-16850023d4a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:05.470327 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.470144    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vhw9\" (UniqueName: \"kubernetes.io/projected/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kube-api-access-6vhw9\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.470327 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.470165    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.470327 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.470179    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.470327 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.470192    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.470327 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.470204    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.470327 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.470216    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.470613 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.470587    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-home" (OuterVolumeSpecName: "home") pod "d0904148-f213-45f5-b3df-16850023d4a6" (UID: "d0904148-f213-45f5-b3df-16850023d4a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:05.471900 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.471867    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0904148-f213-45f5-b3df-16850023d4a6-kube-api-access-jj2sh" (OuterVolumeSpecName: "kube-api-access-jj2sh") pod "d0904148-f213-45f5-b3df-16850023d4a6" (UID: "d0904148-f213-45f5-b3df-16850023d4a6"). InnerVolumeSpecName "kube-api-access-jj2sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:17:05.472036 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.472012    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-dshm" (OuterVolumeSpecName: "dshm") pod "d0904148-f213-45f5-b3df-16850023d4a6" (UID: "d0904148-f213-45f5-b3df-16850023d4a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:05.472192 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.472176    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0904148-f213-45f5-b3df-16850023d4a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d0904148-f213-45f5-b3df-16850023d4a6" (UID: "d0904148-f213-45f5-b3df-16850023d4a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:17:05.483373 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.483352    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d0904148-f213-45f5-b3df-16850023d4a6" (UID: "d0904148-f213-45f5-b3df-16850023d4a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:05.549831 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.549739    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d0904148-f213-45f5-b3df-16850023d4a6" (UID: "d0904148-f213-45f5-b3df-16850023d4a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:05.571128 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.571101    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.571128 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.571127    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.571260 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.571138    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj2sh\" (UniqueName: \"kubernetes.io/projected/d0904148-f213-45f5-b3df-16850023d4a6-kube-api-access-jj2sh\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.571260 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.571146    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.571260 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.571155    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0904148-f213-45f5-b3df-16850023d4a6-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.571260 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.571162    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0904148-f213-45f5-b3df-16850023d4a6-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:05.767217 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.767183    2581 generic.go:356] "Generic (PLEG): container finished" podID="d0904148-f213-45f5-b3df-16850023d4a6" containerID="dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53" exitCode=0
Jul 09 10:17:05.767217 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.767211    2581 generic.go:356] "Generic (PLEG): container finished" podID="d0904148-f213-45f5-b3df-16850023d4a6" containerID="f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11" exitCode=0
Jul 09 10:17:05.767447 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.767235    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"
Jul 09 10:17:05.767447 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.767260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" event={"ID":"d0904148-f213-45f5-b3df-16850023d4a6","Type":"ContainerDied","Data":"dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53"}
Jul 09 10:17:05.767447 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.767315    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" event={"ID":"d0904148-f213-45f5-b3df-16850023d4a6","Type":"ContainerDied","Data":"f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11"}
Jul 09 10:17:05.767447 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.767342    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672" event={"ID":"d0904148-f213-45f5-b3df-16850023d4a6","Type":"ContainerDied","Data":"517f578b745f5527888d6c44c1971040a1f1423b792002fb6ec74f393770ddc0"}
Jul 09 10:17:05.767447 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.767373    2581 scope.go:117] "RemoveContainer" containerID="dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53"
Jul 09 10:17:05.768814 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.768787    2581 generic.go:356] "Generic (PLEG): container finished" podID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerID="e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235" exitCode=0
Jul 09 10:17:05.768945 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.768827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" event={"ID":"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064","Type":"ContainerDied","Data":"e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235"}
Jul 09 10:17:05.768945 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.768846    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"
Jul 09 10:17:05.768945 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.768872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl" event={"ID":"3c5c820c-df5b-4f8a-8d72-dc8f1db2c064","Type":"ContainerDied","Data":"45b9254b017ccc9bc5e5c791d0247b4cf3c7e107a8f205b0426c382718d46c87"}
Jul 09 10:17:05.776704 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.776687    2581 scope.go:117] "RemoveContainer" containerID="47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7"
Jul 09 10:17:05.784536 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.784520    2581 scope.go:117] "RemoveContainer" containerID="f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11"
Jul 09 10:17:05.791604 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.791581    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"]
Jul 09 10:17:05.793043 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.793028    2581 scope.go:117] "RemoveContainer" containerID="dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53"
Jul 09 10:17:05.793288 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:17:05.793271    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53\": container with ID starting with dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53 not found: ID does not exist" containerID="dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53"
Jul 09 10:17:05.793343 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.793296    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53"} err="failed to get container status \"dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53\": rpc error: code = NotFound desc = could not find container \"dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53\": container with ID starting with dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53 not found: ID does not exist"
Jul 09 10:17:05.793343 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.793312    2581 scope.go:117] "RemoveContainer" containerID="47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7"
Jul 09 10:17:05.793555 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:17:05.793539    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7\": container with ID starting with 47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7 not found: ID does not exist" containerID="47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7"
Jul 09 10:17:05.793604 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.793561    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7"} err="failed to get container status \"47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7\": rpc error: code = NotFound desc = could not find container \"47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7\": container with ID starting with 47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7 not found: ID does not exist"
Jul 09 10:17:05.793604 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.793577    2581 scope.go:117] "RemoveContainer" containerID="f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11"
Jul 09 10:17:05.793773 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:17:05.793757    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11\": container with ID starting with f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11 not found: ID does not exist" containerID="f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11"
Jul 09 10:17:05.793819 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.793779    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11"} err="failed to get container status \"f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11\": rpc error: code = NotFound desc = could not find container \"f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11\": container with ID starting with f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11 not found: ID does not exist"
Jul 09 10:17:05.793819 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.793794    2581 scope.go:117] "RemoveContainer" containerID="dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53"
Jul 09 10:17:05.794050 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.794034    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53"} err="failed to get container status \"dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53\": rpc error: code = NotFound desc = could not find container \"dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53\": container with ID starting with dbd979c95237f9894c5d0a5573f70937368336604e2327320f0215e80edd0c53 not found: ID does not exist"
Jul 09 10:17:05.794100 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.794051    2581 scope.go:117] "RemoveContainer" containerID="47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7"
Jul 09 10:17:05.794276 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.794258    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7"} err="failed to get container status \"47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7\": rpc error: code = NotFound desc = could not find container \"47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7\": container with ID starting with 47214fd9435ae3f432a1eddb34cc148cf550e37d508b67e218e570dee17495b7 not found: ID does not exist"
Jul 09 10:17:05.794341 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.794279    2581 scope.go:117] "RemoveContainer" containerID="f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11"
Jul 09 10:17:05.794500 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.794480    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11"} err="failed to get container status \"f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11\": rpc error: code = NotFound desc = could not find container \"f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11\": container with ID starting with f36ef7feecc487b6d69e38e577827bb501de51022dc0f28cea3596a5a4ab1c11 not found: ID does not exist"
Jul 09 10:17:05.794571 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.794502    2581 scope.go:117] "RemoveContainer" containerID="e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235"
Jul 09 10:17:05.798072 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.798051    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5cxkcjl"]
Jul 09 10:17:05.801686 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.801669    2581 scope.go:117] "RemoveContainer" containerID="04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53"
Jul 09 10:17:05.808727 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.808713    2581 scope.go:117] "RemoveContainer" containerID="e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235"
Jul 09 10:17:05.808944 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:17:05.808929    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235\": container with ID starting with e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235 not found: ID does not exist" containerID="e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235"
Jul 09 10:17:05.808997 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.808950    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235"} err="failed to get container status \"e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235\": rpc error: code = NotFound desc = could not find container \"e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235\": container with ID starting with e5920f1a0a112939d2f505e35fde98e34195459cf6097b582a1e571a458ce235 not found: ID does not exist"
Jul 09 10:17:05.808997 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.808964    2581 scope.go:117] "RemoveContainer" containerID="04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53"
Jul 09 10:17:05.809161 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:17:05.809147    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53\": container with ID starting with 04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53 not found: ID does not exist" containerID="04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53"
Jul 09 10:17:05.809198 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.809164    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53"} err="failed to get container status \"04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53\": rpc error: code = NotFound desc = could not find container \"04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53\": container with ID starting with 04e3d909d496eb27d4d5e872ad43bab68536fed438b917722bc0c5f69fba8f53 not found: ID does not exist"
Jul 09 10:17:05.814557 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.814530    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"]
Jul 09 10:17:05.819558 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.819535    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-684f94cfb-k7672"]
Jul 09 10:17:05.846463 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:05.846435    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:17:06.228298 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:06.228263    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" path="/var/lib/kubelet/pods/3c5c820c-df5b-4f8a-8d72-dc8f1db2c064/volumes"
Jul 09 10:17:06.228769 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:06.228663    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0904148-f213-45f5-b3df-16850023d4a6" path="/var/lib/kubelet/pods/d0904148-f213-45f5-b3df-16850023d4a6/volumes"
Jul 09 10:17:14.752637 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752606    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"]
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752908    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752921    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752930    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="storage-initializer"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752936    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="storage-initializer"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752954    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="llm-d-routing-sidecar"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752960    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="llm-d-routing-sidecar"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752965    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="storage-initializer"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752970    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="storage-initializer"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752977    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.752982    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.753022    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="llm-d-routing-sidecar"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.753030    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0904148-f213-45f5-b3df-16850023d4a6" containerName="main"
Jul 09 10:17:14.753078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.753037    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c5c820c-df5b-4f8a-8d72-dc8f1db2c064" containerName="main"
Jul 09 10:17:14.757606 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.757588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.760236 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.760214    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 09 10:17:14.761123 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.761103    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-ng6dp\""
Jul 09 10:17:14.762572 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.762552    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"]
Jul 09 10:17:14.765690 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.765674    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.772401 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.772381    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"]
Jul 09 10:17:14.779082 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.779059    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"]
Jul 09 10:17:14.841807 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.841782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.841807 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.841814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgck\" (UniqueName: \"kubernetes.io/projected/a7c7f407-642e-4983-b263-16c9e48fd16b-kube-api-access-4sgck\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.842025 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.841837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c7f407-642e-4983-b263-16c9e48fd16b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.842025 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.841866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.842025 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.841906    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-home\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.842025 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.841942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.842025 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.841963    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f169b28d-dac0-4e56-8617-d2d16635429b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.842025 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.841981    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.842025 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.842024    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.842240 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.842049    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.842240 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.842068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.842240 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.842102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-dshm\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.842240 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.842117    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vq56\" (UniqueName: \"kubernetes.io/projected/f169b28d-dac0-4e56-8617-d2d16635429b-kube-api-access-4vq56\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.842240 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.842133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.943410 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.943410 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f169b28d-dac0-4e56-8617-d2d16635429b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.943673 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.943673 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.943673 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.943673 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.943673 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-dshm\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.943977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vq56\" (UniqueName: \"kubernetes.io/projected/f169b28d-dac0-4e56-8617-d2d16635429b-kube-api-access-4vq56\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.943977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.943977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.943977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgck\" (UniqueName: \"kubernetes.io/projected/a7c7f407-642e-4983-b263-16c9e48fd16b-kube-api-access-4sgck\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.943977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943809    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.943977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c7f407-642e-4983-b263-16c9e48fd16b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.943977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943825    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.943977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.943977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-home\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.943977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.943946    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-model-cache\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.944455 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.944116    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.944455 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.944340    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.944455 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.944351    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.944455 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.944421    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-home\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.944455 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.944439    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.946129 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.946097    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-dshm\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.946250 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.946236    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.946376 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.946353    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f169b28d-dac0-4e56-8617-d2d16635429b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.946525 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.946510    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c7f407-642e-4983-b263-16c9e48fd16b-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:14.952124 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.952105    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vq56\" (UniqueName: \"kubernetes.io/projected/f169b28d-dac0-4e56-8617-d2d16635429b-kube-api-access-4vq56\") pod \"custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:14.952426 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:14.952404    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgck\" (UniqueName: \"kubernetes.io/projected/a7c7f407-642e-4983-b263-16c9e48fd16b-kube-api-access-4sgck\") pod \"custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:15.067471 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.067390    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:15.074672 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.074637    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:15.229912 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.229858    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"]
Jul 09 10:17:15.236722 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:17:15.236695    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf169b28d_dac0_4e56_8617_d2d16635429b.slice/crio-2dd9190e8ba617105931dbcd600101c72b525a030cc405162bf2aed6760b7657 WatchSource:0}: Error finding container 2dd9190e8ba617105931dbcd600101c72b525a030cc405162bf2aed6760b7657: Status 404 returned error can't find the container with id 2dd9190e8ba617105931dbcd600101c72b525a030cc405162bf2aed6760b7657
Jul 09 10:17:15.248087 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.248068    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"]
Jul 09 10:17:15.250337 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:17:15.250317    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7c7f407_642e_4983_b263_16c9e48fd16b.slice/crio-08eec6df24aa4778299b49a19ebc0305bde11524fc51e99829e1ae52f932eba9 WatchSource:0}: Error finding container 08eec6df24aa4778299b49a19ebc0305bde11524fc51e99829e1ae52f932eba9: Status 404 returned error can't find the container with id 08eec6df24aa4778299b49a19ebc0305bde11524fc51e99829e1ae52f932eba9
Jul 09 10:17:15.810457 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.810417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" event={"ID":"a7c7f407-642e-4983-b263-16c9e48fd16b","Type":"ContainerStarted","Data":"0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065"}
Jul 09 10:17:15.810916 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.810480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" event={"ID":"a7c7f407-642e-4983-b263-16c9e48fd16b","Type":"ContainerStarted","Data":"08eec6df24aa4778299b49a19ebc0305bde11524fc51e99829e1ae52f932eba9"}
Jul 09 10:17:15.811827 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.811808    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" event={"ID":"f169b28d-dac0-4e56-8617-d2d16635429b","Type":"ContainerStarted","Data":"51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee"}
Jul 09 10:17:15.811899 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.811846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" event={"ID":"f169b28d-dac0-4e56-8617-d2d16635429b","Type":"ContainerStarted","Data":"2dd9190e8ba617105931dbcd600101c72b525a030cc405162bf2aed6760b7657"}
Jul 09 10:17:15.811946 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.811930    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:15.855771 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.855747    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:17:15.864314 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:15.864286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:17:16.819329 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:16.819278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" event={"ID":"f169b28d-dac0-4e56-8617-d2d16635429b","Type":"ContainerStarted","Data":"5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8"}
Jul 09 10:17:18.830737 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:18.830701    2581 generic.go:356] "Generic (PLEG): container finished" podID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerID="0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065" exitCode=0
Jul 09 10:17:18.831133 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:18.830774    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" event={"ID":"a7c7f407-642e-4983-b263-16c9e48fd16b","Type":"ContainerDied","Data":"0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065"}
Jul 09 10:17:19.836435 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:19.836403    2581 generic.go:356] "Generic (PLEG): container finished" podID="f169b28d-dac0-4e56-8617-d2d16635429b" containerID="5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8" exitCode=0
Jul 09 10:17:19.836875 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:19.836472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" event={"ID":"f169b28d-dac0-4e56-8617-d2d16635429b","Type":"ContainerDied","Data":"5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8"}
Jul 09 10:17:19.838537 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:19.838516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" event={"ID":"a7c7f407-642e-4983-b263-16c9e48fd16b","Type":"ContainerStarted","Data":"702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12"}
Jul 09 10:17:19.881789 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:19.881746    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podStartSLOduration=2.555027407 podStartE2EDuration="5.881730456s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.326703049s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:17:19.879723708 +0000 UTC m=+2632.204685992" watchObservedRunningTime="2026-07-09 10:17:19.881730456 +0000 UTC m=+2632.206692740"
Jul 09 10:17:20.845341 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:20.845305    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" event={"ID":"f169b28d-dac0-4e56-8617-d2d16635429b","Type":"ContainerStarted","Data":"2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0"}
Jul 09 10:17:20.875497 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:20.875438    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podStartSLOduration=3.597464479 podStartE2EDuration="6.875415565s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.277951086s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:17:20.872950518 +0000 UTC m=+2633.197912805" watchObservedRunningTime="2026-07-09 10:17:20.875415565 +0000 UTC m=+2633.200377854"
Jul 09 10:17:25.068599 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:25.068551    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:25.068599 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:25.068619    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:25.070017 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:25.069970    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:17:25.075196 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:25.075170    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:25.075326 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:25.075225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:17:25.076667 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:25.076632    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:17:25.082904 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:25.082869    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:17:25.461147 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:25.461105    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"]
Jul 09 10:17:35.068670 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.068623    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:17:35.075426 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.075389    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:17:35.791433 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.791400    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"]
Jul 09 10:17:35.801379 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.801351    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:35.805921 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.805878    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"]
Jul 09 10:17:35.934584 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.934548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tls-certs\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:35.934584 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.934583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-model-cache\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:35.934855 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.934621    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-dshm\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:35.934855 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.934701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kserve-provision-location\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:35.934855 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.934732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-home\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:35.934855 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.934818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tmp-dir\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:35.935184 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:35.934859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twztr\" (UniqueName: \"kubernetes.io/projected/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kube-api-access-twztr\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036162 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tmp-dir\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036162 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twztr\" (UniqueName: \"kubernetes.io/projected/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kube-api-access-twztr\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036417 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tls-certs\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036417 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-model-cache\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036417 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-dshm\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036570 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kserve-provision-location\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036570 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036465    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-home\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036590    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tmp-dir\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036606    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-model-cache\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036899 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036850    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-home\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.036993 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.036972    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kserve-provision-location\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.038699 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.038673    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-dshm\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.039344 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.039321    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tls-certs\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.045218 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.045147    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twztr\" (UniqueName: \"kubernetes.io/projected/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kube-api-access-twztr\") pod \"stop-feature-test-kserve-5446b4db4b-2zt2p\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.113938 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.113855    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:36.262538 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.262514    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"]
Jul 09 10:17:36.265878 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:17:36.265844    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5097fff_0165_4348_a7b9_343bbe2d5dcd.slice/crio-1a39c09f2f70e095b09c5aaadd864ee942e6432a26756f8a676d49ccf2ed206f WatchSource:0}: Error finding container 1a39c09f2f70e095b09c5aaadd864ee942e6432a26756f8a676d49ccf2ed206f: Status 404 returned error can't find the container with id 1a39c09f2f70e095b09c5aaadd864ee942e6432a26756f8a676d49ccf2ed206f
Jul 09 10:17:36.921382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.921336    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" event={"ID":"f5097fff-0165-4348-a7b9-343bbe2d5dcd","Type":"ContainerStarted","Data":"2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4"}
Jul 09 10:17:36.921560 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:36.921400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" event={"ID":"f5097fff-0165-4348-a7b9-343bbe2d5dcd","Type":"ContainerStarted","Data":"1a39c09f2f70e095b09c5aaadd864ee942e6432a26756f8a676d49ccf2ed206f"}
Jul 09 10:17:39.938507 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:39.938421    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerID="2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4" exitCode=0
Jul 09 10:17:39.938507 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:39.938484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" event={"ID":"f5097fff-0165-4348-a7b9-343bbe2d5dcd","Type":"ContainerDied","Data":"2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4"}
Jul 09 10:17:40.484803 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:40.484745    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" containerID="cri-o://8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5" gracePeriod=45
Jul 09 10:17:40.856665 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:40.856572    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 09 10:17:40.947321 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:40.947281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" event={"ID":"f5097fff-0165-4348-a7b9-343bbe2d5dcd","Type":"ContainerStarted","Data":"8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c"}
Jul 09 10:17:40.971409 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:40.971338    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podStartSLOduration=2.833367452 podStartE2EDuration="5.971316735s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.137949283s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:17:40.968867309 +0000 UTC m=+2653.293829595" watchObservedRunningTime="2026-07-09 10:17:40.971316735 +0000 UTC m=+2653.296279020"
Jul 09 10:17:41.114874 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.114785    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:41.266619 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.266592    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:17:41.387029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.386931    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-model-cache\") pod \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") "
Jul 09 10:17:41.387029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.386981    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tls-certs\") pod \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") "
Jul 09 10:17:41.387029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.387003    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmb5q\" (UniqueName: \"kubernetes.io/projected/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kube-api-access-xmb5q\") pod \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") "
Jul 09 10:17:41.387303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.387044    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kserve-provision-location\") pod \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") "
Jul 09 10:17:41.387303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.387068    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-dshm\") pod \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") "
Jul 09 10:17:41.387303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.387106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-home\") pod \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") "
Jul 09 10:17:41.387303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.387136    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tmp-dir\") pod \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\" (UID: \"0be8d58c-0e57-463d-8e0d-1bb3661481bd\") "
Jul 09 10:17:41.388995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.387586    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-model-cache" (OuterVolumeSpecName: "model-cache") pod "0be8d58c-0e57-463d-8e0d-1bb3661481bd" (UID: "0be8d58c-0e57-463d-8e0d-1bb3661481bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:41.388995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.388044    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-home" (OuterVolumeSpecName: "home") pod "0be8d58c-0e57-463d-8e0d-1bb3661481bd" (UID: "0be8d58c-0e57-463d-8e0d-1bb3661481bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:41.390530 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.390480    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-dshm" (OuterVolumeSpecName: "dshm") pod "0be8d58c-0e57-463d-8e0d-1bb3661481bd" (UID: "0be8d58c-0e57-463d-8e0d-1bb3661481bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:41.390530 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.390487    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kube-api-access-xmb5q" (OuterVolumeSpecName: "kube-api-access-xmb5q") pod "0be8d58c-0e57-463d-8e0d-1bb3661481bd" (UID: "0be8d58c-0e57-463d-8e0d-1bb3661481bd"). InnerVolumeSpecName "kube-api-access-xmb5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:17:41.402403 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.402375    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0be8d58c-0e57-463d-8e0d-1bb3661481bd" (UID: "0be8d58c-0e57-463d-8e0d-1bb3661481bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:41.403312 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.403274    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0be8d58c-0e57-463d-8e0d-1bb3661481bd" (UID: "0be8d58c-0e57-463d-8e0d-1bb3661481bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:17:41.476703 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.476650    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0be8d58c-0e57-463d-8e0d-1bb3661481bd" (UID: "0be8d58c-0e57-463d-8e0d-1bb3661481bd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:17:41.488446 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.488416    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:41.488446 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.488447    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:41.488615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.488459    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:41.488615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.488470    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:41.488615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.488482    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0be8d58c-0e57-463d-8e0d-1bb3661481bd-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:41.488615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.488492    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0be8d58c-0e57-463d-8e0d-1bb3661481bd-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:41.488615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.488504    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmb5q\" (UniqueName: \"kubernetes.io/projected/0be8d58c-0e57-463d-8e0d-1bb3661481bd-kube-api-access-xmb5q\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:17:41.955029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.954990    2581 generic.go:356] "Generic (PLEG): container finished" podID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerID="8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5" exitCode=0
Jul 09 10:17:41.955502 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.955046    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"
Jul 09 10:17:41.955502 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.955073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" event={"ID":"0be8d58c-0e57-463d-8e0d-1bb3661481bd","Type":"ContainerDied","Data":"8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5"}
Jul 09 10:17:41.955502 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.955143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4" event={"ID":"0be8d58c-0e57-463d-8e0d-1bb3661481bd","Type":"ContainerDied","Data":"42a95f152811c385707cd44bca23f302a991c29131b69acc6d675463c51e2c77"}
Jul 09 10:17:41.955502 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.955176    2581 scope.go:117] "RemoveContainer" containerID="8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5"
Jul 09 10:17:41.966806 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.966789    2581 scope.go:117] "RemoveContainer" containerID="d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9"
Jul 09 10:17:41.976596 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.976570    2581 scope.go:117] "RemoveContainer" containerID="8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5"
Jul 09 10:17:41.976881 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:17:41.976854    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5\": container with ID starting with 8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5 not found: ID does not exist" containerID="8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5"
Jul 09 10:17:41.977002 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.976910    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5"} err="failed to get container status \"8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5\": rpc error: code = NotFound desc = could not find container \"8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5\": container with ID starting with 8801abaee9774169eac7850159894d5efd0eb86eba604d03e608fa248a2cf1e5 not found: ID does not exist"
Jul 09 10:17:41.977002 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.976937    2581 scope.go:117] "RemoveContainer" containerID="d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9"
Jul 09 10:17:41.977209 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:17:41.977190    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9\": container with ID starting with d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9 not found: ID does not exist" containerID="d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9"
Jul 09 10:17:41.977274 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.977215    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9"} err="failed to get container status \"d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9\": rpc error: code = NotFound desc = could not find container \"d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9\": container with ID starting with d9afd10e49972bd0424ac597f66302829a2c005d2f07028ece08c0593ecb29b9 not found: ID does not exist"
Jul 09 10:17:41.980996 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.980972    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"]
Jul 09 10:17:41.984538 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:41.984519    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-x7nl4"]
Jul 09 10:17:42.229749 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:42.229652    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" path="/var/lib/kubelet/pods/0be8d58c-0e57-463d-8e0d-1bb3661481bd/volumes"
Jul 09 10:17:45.068016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:45.067973    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:17:45.075781 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:45.075745    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:17:46.114636 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:46.114599    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:17:46.116564 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:46.116526    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:17:55.068189 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:55.068132    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:17:55.075835 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:55.075792    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:17:56.115396 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:17:56.115351    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:18:05.068016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:05.067970    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:18:05.075735 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:05.075690    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:18:06.114528 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:06.114489    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:18:15.068376 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:15.068317    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:18:15.075987 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:15.075954    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:18:16.114363 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:16.114314    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:18:25.067950 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:25.067843    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:18:25.075213 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:25.075177    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:18:26.115064 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:26.115009    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:18:30.140279 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:30.140247    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:18:30.144463 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:30.144441    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:18:35.068538 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:35.068491    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:18:35.075642 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:35.075609    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:18:36.115129 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:36.115080    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:18:45.068346 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:45.068056    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:18:45.075801 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:45.075766    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:18:46.114977 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:46.114933    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:18:55.068433 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:55.068381    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:18:55.075294 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:55.075263    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:18:56.114428 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:18:56.114392    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:19:05.068119 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:05.068069    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:19:05.075820 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:05.075790    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:19:06.114822 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:06.114784    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:19:15.068562 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:15.068514    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:19:15.075798 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:15.075771    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:19:16.114510 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:16.114464    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:19:25.068598 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:25.068549    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:19:25.075614 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:25.075584    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:19:26.115316 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:26.115267    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:19:35.068149 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:35.068099    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:19:35.075622 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:35.075585    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:19:36.115014 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:36.114962    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:19:45.068359 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:45.068318    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:19:45.074938 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:45.074903    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:19:46.114991 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:46.114946    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:19:55.068146 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:55.067915    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:19:55.075177 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:55.075144    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:19:56.114444 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:19:56.114399    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:20:05.068710 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:05.068668    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:20:05.075569 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:05.075537    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:20:06.114412 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:06.114366    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:20:15.068685 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:15.068641    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8001/health\": dial tcp 10.133.0.48:8001: connect: connection refused"
Jul 09 10:20:15.075208 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:15.075175    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.49:8000/health\": dial tcp 10.133.0.49:8000: connect: connection refused"
Jul 09 10:20:16.114736 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:16.114691    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:20:25.080117 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:25.080074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:20:25.086780 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:25.086748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:20:25.095351 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:25.095326    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:20:25.097527 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:25.097508    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:20:26.115248 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:26.115207    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:20:36.124810 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:36.124769    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:20:36.133477 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:36.133447    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:20:37.389669 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:37.389635    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"]
Jul 09 10:20:52.737377 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:52.737332    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" containerID="cri-o://8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c" gracePeriod=45
Jul 09 10:20:53.126513 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.126414    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 09 10:20:53.482727 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.482703    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:20:53.637775 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.637739    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tmp-dir\") pod \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") "
Jul 09 10:20:53.637995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.637815    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tls-certs\") pod \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") "
Jul 09 10:20:53.637995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.637861    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-dshm\") pod \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") "
Jul 09 10:20:53.637995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.637917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kserve-provision-location\") pod \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") "
Jul 09 10:20:53.637995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.637945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-home\") pod \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") "
Jul 09 10:20:53.637995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.637978    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twztr\" (UniqueName: \"kubernetes.io/projected/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kube-api-access-twztr\") pod \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") "
Jul 09 10:20:53.638284 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.638030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-model-cache\") pod \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\" (UID: \"f5097fff-0165-4348-a7b9-343bbe2d5dcd\") "
Jul 09 10:20:53.638487 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.638460    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-model-cache" (OuterVolumeSpecName: "model-cache") pod "f5097fff-0165-4348-a7b9-343bbe2d5dcd" (UID: "f5097fff-0165-4348-a7b9-343bbe2d5dcd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:20:53.638925 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.638866    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-home" (OuterVolumeSpecName: "home") pod "f5097fff-0165-4348-a7b9-343bbe2d5dcd" (UID: "f5097fff-0165-4348-a7b9-343bbe2d5dcd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:20:53.640268 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.640242    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f5097fff-0165-4348-a7b9-343bbe2d5dcd" (UID: "f5097fff-0165-4348-a7b9-343bbe2d5dcd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:20:53.640389 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.640369    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-dshm" (OuterVolumeSpecName: "dshm") pod "f5097fff-0165-4348-a7b9-343bbe2d5dcd" (UID: "f5097fff-0165-4348-a7b9-343bbe2d5dcd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:20:53.641021 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.640990    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kube-api-access-twztr" (OuterVolumeSpecName: "kube-api-access-twztr") pod "f5097fff-0165-4348-a7b9-343bbe2d5dcd" (UID: "f5097fff-0165-4348-a7b9-343bbe2d5dcd"). InnerVolumeSpecName "kube-api-access-twztr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:20:53.653775 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.653702    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f5097fff-0165-4348-a7b9-343bbe2d5dcd" (UID: "f5097fff-0165-4348-a7b9-343bbe2d5dcd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:20:53.699717 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.699674    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5097fff-0165-4348-a7b9-343bbe2d5dcd" (UID: "f5097fff-0165-4348-a7b9-343bbe2d5dcd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:20:53.739071 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.739038    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:20:53.739071 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.739064    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:20:53.739071 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.739076    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:20:53.739492 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.739088    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:20:53.739492 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.739103    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twztr\" (UniqueName: \"kubernetes.io/projected/f5097fff-0165-4348-a7b9-343bbe2d5dcd-kube-api-access-twztr\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:20:53.739492 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.739114    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:20:53.739492 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.739122    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5097fff-0165-4348-a7b9-343bbe2d5dcd-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:20:53.776763 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.776731    2581 generic.go:356] "Generic (PLEG): container finished" podID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerID="8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c" exitCode=0
Jul 09 10:20:53.776946 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.776785    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"
Jul 09 10:20:53.776946 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.776784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" event={"ID":"f5097fff-0165-4348-a7b9-343bbe2d5dcd","Type":"ContainerDied","Data":"8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c"}
Jul 09 10:20:53.776946 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.776935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p" event={"ID":"f5097fff-0165-4348-a7b9-343bbe2d5dcd","Type":"ContainerDied","Data":"1a39c09f2f70e095b09c5aaadd864ee942e6432a26756f8a676d49ccf2ed206f"}
Jul 09 10:20:53.777057 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.776965    2581 scope.go:117] "RemoveContainer" containerID="8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c"
Jul 09 10:20:53.786565 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.786514    2581 scope.go:117] "RemoveContainer" containerID="2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4"
Jul 09 10:20:53.795501 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.795484    2581 scope.go:117] "RemoveContainer" containerID="8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c"
Jul 09 10:20:53.795753 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:20:53.795733    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c\": container with ID starting with 8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c not found: ID does not exist" containerID="8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c"
Jul 09 10:20:53.795803 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.795761    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c"} err="failed to get container status \"8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c\": rpc error: code = NotFound desc = could not find container \"8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c\": container with ID starting with 8a9879c279973a505c8dc75ae70f6044fdfc5d51065801979bd8626d5d2ed63c not found: ID does not exist"
Jul 09 10:20:53.795803 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.795779    2581 scope.go:117] "RemoveContainer" containerID="2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4"
Jul 09 10:20:53.796019 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:20:53.795996    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4\": container with ID starting with 2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4 not found: ID does not exist" containerID="2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4"
Jul 09 10:20:53.796075 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.796025    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4"} err="failed to get container status \"2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4\": rpc error: code = NotFound desc = could not find container \"2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4\": container with ID starting with 2412e52f48e74e5705d9607f088519fde3fc7f80be7f65b8822b194ffbf202e4 not found: ID does not exist"
Jul 09 10:20:53.799594 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.799573    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"]
Jul 09 10:20:53.803309 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:53.803286    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-5446b4db4b-2zt2p"]
Jul 09 10:20:54.228266 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:20:54.228234    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" path="/var/lib/kubelet/pods/f5097fff-0165-4348-a7b9-343bbe2d5dcd/volumes"
Jul 09 10:21:16.093781 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.093747    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"]
Jul 09 10:21:16.094269 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.094066    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main"
Jul 09 10:21:16.094269 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.094078    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main"
Jul 09 10:21:16.094269 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.094088    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main"
Jul 09 10:21:16.094269 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.094093    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main"
Jul 09 10:21:16.094269 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.094100    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="storage-initializer"
Jul 09 10:21:16.094269 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.094105    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="storage-initializer"
Jul 09 10:21:16.094269 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.094117    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="storage-initializer"
Jul 09 10:21:16.094269 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.094123    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="storage-initializer"
Jul 09 10:21:16.094269 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.094171    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5097fff-0165-4348-a7b9-343bbe2d5dcd" containerName="main"
Jul 09 10:21:16.094269 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.094179    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0be8d58c-0e57-463d-8e0d-1bb3661481bd" containerName="main"
Jul 09 10:21:16.096215 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.096198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.099970 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.099950    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 10:21:16.100079 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.099951    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 09 10:21:16.111013 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.110989    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"]
Jul 09 10:21:16.128605 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.128573    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd242324-7a6e-4633-82f3-190a80d1cdb9-cabundle-cert\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.128709 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.128614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-tmp-dir\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.128709 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.128680    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.128788 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.128707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqsdm\" (UniqueName: \"kubernetes.io/projected/cd242324-7a6e-4633-82f3-190a80d1cdb9-kube-api-access-cqsdm\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.128823 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.128786    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-home\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.128857 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.128827    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-dshm\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.128912 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.128861    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-model-cache\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.128952 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.128912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd242324-7a6e-4633-82f3-190a80d1cdb9-tls-certs\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.229268 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd242324-7a6e-4633-82f3-190a80d1cdb9-cabundle-cert\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.229459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-tmp-dir\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.229459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.229459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqsdm\" (UniqueName: \"kubernetes.io/projected/cd242324-7a6e-4633-82f3-190a80d1cdb9-kube-api-access-cqsdm\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.229459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-home\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.229459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-dshm\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.229459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-model-cache\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.229781 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229474    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd242324-7a6e-4633-82f3-190a80d1cdb9-tls-certs\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.229970 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229936    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-tmp-dir\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.230069 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229967    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.230069 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.229986    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-home\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.230069 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.230029    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd242324-7a6e-4633-82f3-190a80d1cdb9-cabundle-cert\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.230172 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.230077    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-model-cache\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.231644 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.231616    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-dshm\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.231956 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.231927    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd242324-7a6e-4633-82f3-190a80d1cdb9-tls-certs\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.237516 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.237488    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqsdm\" (UniqueName: \"kubernetes.io/projected/cd242324-7a6e-4633-82f3-190a80d1cdb9-kube-api-access-cqsdm\") pod \"lora-single-adapter-test-kserve-64b48975c9-spbbv\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.406252 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.406158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:16.545177 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.545151    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"]
Jul 09 10:21:16.546664 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:21:16.546638    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd242324_7a6e_4633_82f3_190a80d1cdb9.slice/crio-7c7ac05fdc13f463436fac8ab1391a5399ec7ac7696dc90da31bfbc8cd24d7b4 WatchSource:0}: Error finding container 7c7ac05fdc13f463436fac8ab1391a5399ec7ac7696dc90da31bfbc8cd24d7b4: Status 404 returned error can't find the container with id 7c7ac05fdc13f463436fac8ab1391a5399ec7ac7696dc90da31bfbc8cd24d7b4
Jul 09 10:21:16.870984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.870945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" event={"ID":"cd242324-7a6e-4633-82f3-190a80d1cdb9","Type":"ContainerStarted","Data":"7677b6633e43ff481b0b9aaf851101ecc3fd33b5f2bca5495c62f4447b77f1d4"}
Jul 09 10:21:16.871189 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:16.871006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" event={"ID":"cd242324-7a6e-4633-82f3-190a80d1cdb9","Type":"ContainerStarted","Data":"7c7ac05fdc13f463436fac8ab1391a5399ec7ac7696dc90da31bfbc8cd24d7b4"}
Jul 09 10:21:20.887284 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:20.887250    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerID="7677b6633e43ff481b0b9aaf851101ecc3fd33b5f2bca5495c62f4447b77f1d4" exitCode=0
Jul 09 10:21:20.887655 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:20.887323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" event={"ID":"cd242324-7a6e-4633-82f3-190a80d1cdb9","Type":"ContainerDied","Data":"7677b6633e43ff481b0b9aaf851101ecc3fd33b5f2bca5495c62f4447b77f1d4"}
Jul 09 10:21:21.892950 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:21.892915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" event={"ID":"cd242324-7a6e-4633-82f3-190a80d1cdb9","Type":"ContainerStarted","Data":"39f903d82d68130fd028634a2954acbc602f8283506f7820eeb7e8f33f68ecce"}
Jul 09 10:21:21.917602 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:21.917548    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" podStartSLOduration=2.477777557 podStartE2EDuration="5.917531839s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.439754282s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:21:21.915455141 +0000 UTC m=+2874.240417441" watchObservedRunningTime="2026-07-09 10:21:21.917531839 +0000 UTC m=+2874.242494123"
Jul 09 10:21:22.406398 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:22.406365    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:26.406443 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:26.406411    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:21:26.408124 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:26.408097    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 10:21:36.406984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:36.406879    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 10:21:46.406763 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:46.406721    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 10:21:56.407193 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:21:56.407149    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 10:22:06.406767 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:22:06.406718    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 10:22:16.407258 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:22:16.407214    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 10:22:26.407101 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:22:26.407056    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 10:22:36.416543 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:22:36.416509    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:22:36.424379 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:22:36.424349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:23:30.166566 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:23:30.166537    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:23:30.170579 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:23:30.170555    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:23:46.765561 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:23:46.765525    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"]
Jul 09 10:23:46.775491 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:23:46.775466    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"]
Jul 09 10:24:01.789386 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:01.789342    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main" containerID="cri-o://2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0" gracePeriod=45
Jul 09 10:24:01.797865 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:01.797826    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main" containerID="cri-o://702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12" gracePeriod=45
Jul 09 10:24:02.614398 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.614365    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="llm-d-routing-sidecar" containerID="cri-o://51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee" gracePeriod=45
Jul 09 10:24:02.746735 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.746708    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:24:02.859074 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.859052    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:24:02.877840 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.877819    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c7f407-642e-4983-b263-16c9e48fd16b-tls-certs\") pod \"a7c7f407-642e-4983-b263-16c9e48fd16b\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") "
Jul 09 10:24:02.877941 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.877865    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-home\") pod \"a7c7f407-642e-4983-b263-16c9e48fd16b\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") "
Jul 09 10:24:02.877941 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.877910    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-model-cache\") pod \"a7c7f407-642e-4983-b263-16c9e48fd16b\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") "
Jul 09 10:24:02.878026 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.877948    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-kserve-provision-location\") pod \"a7c7f407-642e-4983-b263-16c9e48fd16b\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") "
Jul 09 10:24:02.878026 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.878007    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4sgck\" (UniqueName: \"kubernetes.io/projected/a7c7f407-642e-4983-b263-16c9e48fd16b-kube-api-access-4sgck\") pod \"a7c7f407-642e-4983-b263-16c9e48fd16b\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") "
Jul 09 10:24:02.878132 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.878029    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-tmp-dir\") pod \"a7c7f407-642e-4983-b263-16c9e48fd16b\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") "
Jul 09 10:24:02.878132 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.878043    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-dshm\") pod \"a7c7f407-642e-4983-b263-16c9e48fd16b\" (UID: \"a7c7f407-642e-4983-b263-16c9e48fd16b\") "
Jul 09 10:24:02.878237 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.878187    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a7c7f407-642e-4983-b263-16c9e48fd16b" (UID: "a7c7f407-642e-4983-b263-16c9e48fd16b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:24:02.878319 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.878301    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:02.878671 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.878642    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-home" (OuterVolumeSpecName: "home") pod "a7c7f407-642e-4983-b263-16c9e48fd16b" (UID: "a7c7f407-642e-4983-b263-16c9e48fd16b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:24:02.880069 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.880041    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7c7f407-642e-4983-b263-16c9e48fd16b-kube-api-access-4sgck" (OuterVolumeSpecName: "kube-api-access-4sgck") pod "a7c7f407-642e-4983-b263-16c9e48fd16b" (UID: "a7c7f407-642e-4983-b263-16c9e48fd16b"). InnerVolumeSpecName "kube-api-access-4sgck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:24:02.880330 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.880308    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7c7f407-642e-4983-b263-16c9e48fd16b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a7c7f407-642e-4983-b263-16c9e48fd16b" (UID: "a7c7f407-642e-4983-b263-16c9e48fd16b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:24:02.880411 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.880368    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-dshm" (OuterVolumeSpecName: "dshm") pod "a7c7f407-642e-4983-b263-16c9e48fd16b" (UID: "a7c7f407-642e-4983-b263-16c9e48fd16b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:24:02.892828 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.892801    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a7c7f407-642e-4983-b263-16c9e48fd16b" (UID: "a7c7f407-642e-4983-b263-16c9e48fd16b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:24:02.954753 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.954717    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a7c7f407-642e-4983-b263-16c9e48fd16b" (UID: "a7c7f407-642e-4983-b263-16c9e48fd16b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:24:02.979528 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.979508    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f169b28d-dac0-4e56-8617-d2d16635429b-tls-certs\") pod \"f169b28d-dac0-4e56-8617-d2d16635429b\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") "
Jul 09 10:24:02.979594 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.979550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-kserve-provision-location\") pod \"f169b28d-dac0-4e56-8617-d2d16635429b\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") "
Jul 09 10:24:02.979594 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.979576    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-dshm\") pod \"f169b28d-dac0-4e56-8617-d2d16635429b\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") "
Jul 09 10:24:02.979664 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.979606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vq56\" (UniqueName: \"kubernetes.io/projected/f169b28d-dac0-4e56-8617-d2d16635429b-kube-api-access-4vq56\") pod \"f169b28d-dac0-4e56-8617-d2d16635429b\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") "
Jul 09 10:24:02.979664 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.979633    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-tmp-dir\") pod \"f169b28d-dac0-4e56-8617-d2d16635429b\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") "
Jul 09 10:24:02.979743 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.979712    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-home\") pod \"f169b28d-dac0-4e56-8617-d2d16635429b\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") "
Jul 09 10:24:02.979789 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.979745    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-model-cache\") pod \"f169b28d-dac0-4e56-8617-d2d16635429b\" (UID: \"f169b28d-dac0-4e56-8617-d2d16635429b\") "
Jul 09 10:24:02.980010 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.979979    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:02.980010 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.980011    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sgck\" (UniqueName: \"kubernetes.io/projected/a7c7f407-642e-4983-b263-16c9e48fd16b-kube-api-access-4sgck\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:02.980182 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.980026    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:02.980182 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.980041    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:02.980182 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.980054    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7c7f407-642e-4983-b263-16c9e48fd16b-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:02.980182 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.980067    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7c7f407-642e-4983-b263-16c9e48fd16b-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:02.980391 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.980203    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f169b28d-dac0-4e56-8617-d2d16635429b" (UID: "f169b28d-dac0-4e56-8617-d2d16635429b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:24:02.980501 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.980474    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-home" (OuterVolumeSpecName: "home") pod "f169b28d-dac0-4e56-8617-d2d16635429b" (UID: "f169b28d-dac0-4e56-8617-d2d16635429b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:24:02.981805 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.981775    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f169b28d-dac0-4e56-8617-d2d16635429b-kube-api-access-4vq56" (OuterVolumeSpecName: "kube-api-access-4vq56") pod "f169b28d-dac0-4e56-8617-d2d16635429b" (UID: "f169b28d-dac0-4e56-8617-d2d16635429b"). InnerVolumeSpecName "kube-api-access-4vq56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:24:02.982032 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.982006    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-dshm" (OuterVolumeSpecName: "dshm") pod "f169b28d-dac0-4e56-8617-d2d16635429b" (UID: "f169b28d-dac0-4e56-8617-d2d16635429b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:24:02.982117 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.982071    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f169b28d-dac0-4e56-8617-d2d16635429b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f169b28d-dac0-4e56-8617-d2d16635429b" (UID: "f169b28d-dac0-4e56-8617-d2d16635429b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:24:02.994993 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:02.994971    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f169b28d-dac0-4e56-8617-d2d16635429b" (UID: "f169b28d-dac0-4e56-8617-d2d16635429b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:24:03.049588 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.049518    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f169b28d-dac0-4e56-8617-d2d16635429b" (UID: "f169b28d-dac0-4e56-8617-d2d16635429b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:24:03.081259 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.081234    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f169b28d-dac0-4e56-8617-d2d16635429b-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:03.081259 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.081259    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:03.081401 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.081267    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:03.081401 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.081277    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vq56\" (UniqueName: \"kubernetes.io/projected/f169b28d-dac0-4e56-8617-d2d16635429b-kube-api-access-4vq56\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:03.081401 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.081289    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:03.081401 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.081298    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:03.081401 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.081305    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f169b28d-dac0-4e56-8617-d2d16635429b-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:24:03.545485 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.545454    2581 generic.go:356] "Generic (PLEG): container finished" podID="f169b28d-dac0-4e56-8617-d2d16635429b" containerID="2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0" exitCode=0
Jul 09 10:24:03.545485 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.545480    2581 generic.go:356] "Generic (PLEG): container finished" podID="f169b28d-dac0-4e56-8617-d2d16635429b" containerID="51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee" exitCode=0
Jul 09 10:24:03.545711 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.545510    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"
Jul 09 10:24:03.545711 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.545540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" event={"ID":"f169b28d-dac0-4e56-8617-d2d16635429b","Type":"ContainerDied","Data":"2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0"}
Jul 09 10:24:03.545711 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.545589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" event={"ID":"f169b28d-dac0-4e56-8617-d2d16635429b","Type":"ContainerDied","Data":"51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee"}
Jul 09 10:24:03.545711 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.545609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56" event={"ID":"f169b28d-dac0-4e56-8617-d2d16635429b","Type":"ContainerDied","Data":"2dd9190e8ba617105931dbcd600101c72b525a030cc405162bf2aed6760b7657"}
Jul 09 10:24:03.545711 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.545635    2581 scope.go:117] "RemoveContainer" containerID="2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0"
Jul 09 10:24:03.547318 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.547293    2581 generic.go:356] "Generic (PLEG): container finished" podID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerID="702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12" exitCode=0
Jul 09 10:24:03.547427 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.547325    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" event={"ID":"a7c7f407-642e-4983-b263-16c9e48fd16b","Type":"ContainerDied","Data":"702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12"}
Jul 09 10:24:03.547427 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.547341    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"
Jul 09 10:24:03.547427 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.547380    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg" event={"ID":"a7c7f407-642e-4983-b263-16c9e48fd16b","Type":"ContainerDied","Data":"08eec6df24aa4778299b49a19ebc0305bde11524fc51e99829e1ae52f932eba9"}
Jul 09 10:24:03.556953 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.556933    2581 scope.go:117] "RemoveContainer" containerID="5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8"
Jul 09 10:24:03.565604 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.565592    2581 scope.go:117] "RemoveContainer" containerID="51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee"
Jul 09 10:24:03.572556 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.572541    2581 scope.go:117] "RemoveContainer" containerID="2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0"
Jul 09 10:24:03.572808 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:24:03.572789    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0\": container with ID starting with 2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0 not found: ID does not exist" containerID="2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0"
Jul 09 10:24:03.572907 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.572816    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0"} err="failed to get container status \"2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0\": rpc error: code = NotFound desc = could not find container \"2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0\": container with ID starting with 2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0 not found: ID does not exist"
Jul 09 10:24:03.572907 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.572832    2581 scope.go:117] "RemoveContainer" containerID="5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8"
Jul 09 10:24:03.573095 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:24:03.573076    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8\": container with ID starting with 5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8 not found: ID does not exist" containerID="5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8"
Jul 09 10:24:03.573153 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.573104    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8"} err="failed to get container status \"5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8\": rpc error: code = NotFound desc = could not find container \"5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8\": container with ID starting with 5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8 not found: ID does not exist"
Jul 09 10:24:03.573153 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.573126    2581 scope.go:117] "RemoveContainer" containerID="51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee"
Jul 09 10:24:03.573331 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:24:03.573312    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee\": container with ID starting with 51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee not found: ID does not exist" containerID="51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee"
Jul 09 10:24:03.573378 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.573337    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee"} err="failed to get container status \"51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee\": rpc error: code = NotFound desc = could not find container \"51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee\": container with ID starting with 51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee not found: ID does not exist"
Jul 09 10:24:03.573378 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.573352    2581 scope.go:117] "RemoveContainer" containerID="2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0"
Jul 09 10:24:03.573514 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.573495    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0"} err="failed to get container status \"2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0\": rpc error: code = NotFound desc = could not find container \"2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0\": container with ID starting with 2e5e643f3a01ebc9ad803bd5b397c553947eea331a8fe18d5bad91ec5571e7a0 not found: ID does not exist"
Jul 09 10:24:03.573554 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.573515    2581 scope.go:117] "RemoveContainer" containerID="5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8"
Jul 09 10:24:03.573682 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.573667    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8"} err="failed to get container status \"5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8\": rpc error: code = NotFound desc = could not find container \"5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8\": container with ID starting with 5f13a068917ae464d7e8dd7e86ed69b649107108d5db9a5d4dc9c00b925fcdc8 not found: ID does not exist"
Jul 09 10:24:03.573726 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.573683    2581 scope.go:117] "RemoveContainer" containerID="51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee"
Jul 09 10:24:03.573935 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.573868    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee"} err="failed to get container status \"51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee\": rpc error: code = NotFound desc = could not find container \"51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee\": container with ID starting with 51ea2e379c630fe5ff77ce8f4616252fb71ff1b1d74580d0f842fc88319ed7ee not found: ID does not exist"
Jul 09 10:24:03.573935 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.573900    2581 scope.go:117] "RemoveContainer" containerID="702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12"
Jul 09 10:24:03.576356 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.576333    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"]
Jul 09 10:24:03.580303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.580279    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-f8f6599f9-fdz56"]
Jul 09 10:24:03.581595 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.581582    2581 scope.go:117] "RemoveContainer" containerID="0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065"
Jul 09 10:24:03.588978 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.588940    2581 scope.go:117] "RemoveContainer" containerID="702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12"
Jul 09 10:24:03.589238 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:24:03.589218    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12\": container with ID starting with 702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12 not found: ID does not exist" containerID="702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12"
Jul 09 10:24:03.589294 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.589241    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12"} err="failed to get container status \"702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12\": rpc error: code = NotFound desc = could not find container \"702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12\": container with ID starting with 702b7b6770d24f5cc9212476a8c2c4ee4f8da8879d73984bac891ba6d4012d12 not found: ID does not exist"
Jul 09 10:24:03.589294 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.589255    2581 scope.go:117] "RemoveContainer" containerID="0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065"
Jul 09 10:24:03.589484 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:24:03.589471    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065\": container with ID starting with 0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065 not found: ID does not exist" containerID="0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065"
Jul 09 10:24:03.589525 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.589486    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065"} err="failed to get container status \"0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065\": rpc error: code = NotFound desc = could not find container \"0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065\": container with ID starting with 0b0090758552a6943912a4d34bc4517234bbca2cca6652797e52e158e8b91065 not found: ID does not exist"
Jul 09 10:24:03.590565 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.590549    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"]
Jul 09 10:24:03.594248 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:03.594229    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-6c9bddf46b-dnsmg"]
Jul 09 10:24:04.227930 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:04.227876    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" path="/var/lib/kubelet/pods/a7c7f407-642e-4983-b263-16c9e48fd16b/volumes"
Jul 09 10:24:04.228358 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:04.228344    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" path="/var/lib/kubelet/pods/f169b28d-dac0-4e56-8617-d2d16635429b/volumes"
Jul 09 10:24:24.574494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574458    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"]
Jul 09 10:24:24.574945 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574850    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="llm-d-routing-sidecar"
Jul 09 10:24:24.574945 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574868    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="llm-d-routing-sidecar"
Jul 09 10:24:24.574945 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574880    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="storage-initializer"
Jul 09 10:24:24.574945 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574916    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="storage-initializer"
Jul 09 10:24:24.574945 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574935    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main"
Jul 09 10:24:24.574945 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574943    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main"
Jul 09 10:24:24.574945 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574957    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="storage-initializer"
Jul 09 10:24:24.575203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574966    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="storage-initializer"
Jul 09 10:24:24.575203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574987    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main"
Jul 09 10:24:24.575203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.574996    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main"
Jul 09 10:24:24.575203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.575075    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="llm-d-routing-sidecar"
Jul 09 10:24:24.575203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.575090    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7c7f407-642e-4983-b263-16c9e48fd16b" containerName="main"
Jul 09 10:24:24.575203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.575098    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f169b28d-dac0-4e56-8617-d2d16635429b" containerName="main"
Jul 09 10:24:24.578426 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.578408    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.581359 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.581339    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 09 10:24:24.582326 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.582302    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-rwgcs\""
Jul 09 10:24:24.600593 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.600570    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"]
Jul 09 10:24:24.604817 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.604792    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"]
Jul 09 10:24:24.608310 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.608291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.631807 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.631780    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"]
Jul 09 10:24:24.644680 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.644779 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644688    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-home\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.644779 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644716    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-model-cache\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.644869 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.644869 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-dshm\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.644972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.644972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pwpc\" (UniqueName: \"kubernetes.io/projected/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kube-api-access-7pwpc\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.644972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.644972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tls-certs\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.644972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.645174 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.644993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6953b3c-6947-4c54-b313-8a4884d866dd-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.645174 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.645019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.645174 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.645041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qpzh\" (UniqueName: \"kubernetes.io/projected/e6953b3c-6947-4c54-b313-8a4884d866dd-kube-api-access-8qpzh\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.645174 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.645060    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-home\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.745774 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.745740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.745774 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.745777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pwpc\" (UniqueName: \"kubernetes.io/projected/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kube-api-access-7pwpc\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.746029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.745801    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.746029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.745816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tls-certs\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.746029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.745836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.746029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.745849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6953b3c-6947-4c54-b313-8a4884d866dd-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.746029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.745873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.746029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.745928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qpzh\" (UniqueName: \"kubernetes.io/projected/e6953b3c-6947-4c54-b313-8a4884d866dd-kube-api-access-8qpzh\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.746356 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-home\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.746356 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.746356 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746190    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-home\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.746356 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746205    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.746356 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-model-cache\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.746356 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746288    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.746356 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746310    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.746356 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-dshm\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.746776 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746514    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.746776 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746538    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.746776 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746537    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-home\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.746776 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746760    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.746960 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746773    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-model-cache\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.746960 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.746815    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-home\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.748308 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.748280    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.748600 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.748574    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tls-certs\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.748702 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.748654    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-dshm\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.748908 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.748871    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6953b3c-6947-4c54-b313-8a4884d866dd-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.759837 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.759811    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qpzh\" (UniqueName: \"kubernetes.io/projected/e6953b3c-6947-4c54-b313-8a4884d866dd-kube-api-access-8qpzh\") pod \"router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:24.760028 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.760009    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pwpc\" (UniqueName: \"kubernetes.io/projected/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kube-api-access-7pwpc\") pod \"router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.888226 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.888151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:24.917194 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:24.917166    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:25.036544 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:24:25.036514    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda214d4f3_aff1_41f9_8f68_0ee9658b9485.slice/crio-41e2a73f3ee8b6891c48bea8e04db3dda55c8fc3cb23dc046ff2d957027bc5bf WatchSource:0}: Error finding container 41e2a73f3ee8b6891c48bea8e04db3dda55c8fc3cb23dc046ff2d957027bc5bf: Status 404 returned error can't find the container with id 41e2a73f3ee8b6891c48bea8e04db3dda55c8fc3cb23dc046ff2d957027bc5bf
Jul 09 10:24:25.036956 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:25.036927    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"]
Jul 09 10:24:25.069264 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:25.069239    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"]
Jul 09 10:24:25.071056 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:24:25.071029    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6953b3c_6947_4c54_b313_8a4884d866dd.slice/crio-44ad13dd3413345c7c80a7b0d068b8ed7e72280b5f0ca533b0775e8d6109946e WatchSource:0}: Error finding container 44ad13dd3413345c7c80a7b0d068b8ed7e72280b5f0ca533b0775e8d6109946e: Status 404 returned error can't find the container with id 44ad13dd3413345c7c80a7b0d068b8ed7e72280b5f0ca533b0775e8d6109946e
Jul 09 10:24:25.631724 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:25.631686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" event={"ID":"a214d4f3-aff1-41f9-8f68-0ee9658b9485","Type":"ContainerStarted","Data":"88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89"}
Jul 09 10:24:25.632201 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:25.631756    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:25.632201 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:25.631780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" event={"ID":"a214d4f3-aff1-41f9-8f68-0ee9658b9485","Type":"ContainerStarted","Data":"41e2a73f3ee8b6891c48bea8e04db3dda55c8fc3cb23dc046ff2d957027bc5bf"}
Jul 09 10:24:25.633253 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:25.633227    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" event={"ID":"e6953b3c-6947-4c54-b313-8a4884d866dd","Type":"ContainerStarted","Data":"bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a"}
Jul 09 10:24:25.639664 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:25.633274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" event={"ID":"e6953b3c-6947-4c54-b313-8a4884d866dd","Type":"ContainerStarted","Data":"44ad13dd3413345c7c80a7b0d068b8ed7e72280b5f0ca533b0775e8d6109946e"}
Jul 09 10:24:26.641662 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:26.641609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" event={"ID":"a214d4f3-aff1-41f9-8f68-0ee9658b9485","Type":"ContainerStarted","Data":"6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3"}
Jul 09 10:24:29.657205 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:29.657174    2581 generic.go:356] "Generic (PLEG): container finished" podID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerID="bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a" exitCode=0
Jul 09 10:24:29.657618 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:29.657251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" event={"ID":"e6953b3c-6947-4c54-b313-8a4884d866dd","Type":"ContainerDied","Data":"bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a"}
Jul 09 10:24:29.659144 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:29.659123    2581 generic.go:356] "Generic (PLEG): container finished" podID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerID="6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3" exitCode=0
Jul 09 10:24:29.659239 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:29.659162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" event={"ID":"a214d4f3-aff1-41f9-8f68-0ee9658b9485","Type":"ContainerDied","Data":"6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3"}
Jul 09 10:24:30.667422 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:30.667371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" event={"ID":"a214d4f3-aff1-41f9-8f68-0ee9658b9485","Type":"ContainerStarted","Data":"b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a"}
Jul 09 10:24:30.669420 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:30.669397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" event={"ID":"e6953b3c-6947-4c54-b313-8a4884d866dd","Type":"ContainerStarted","Data":"aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea"}
Jul 09 10:24:30.697422 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:30.697347    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podStartSLOduration=3.06382288 podStartE2EDuration="6.697325814s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.633502934s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:24:30.695632344 +0000 UTC m=+3063.020594643" watchObservedRunningTime="2026-07-09 10:24:30.697325814 +0000 UTC m=+3063.022288099"
Jul 09 10:24:30.723436 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:30.723384    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podStartSLOduration=2.651513179 podStartE2EDuration="6.723368373s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.071855194s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:24:30.720945783 +0000 UTC m=+3063.045908064" watchObservedRunningTime="2026-07-09 10:24:30.723368373 +0000 UTC m=+3063.048330656"
Jul 09 10:24:34.889197 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:34.889152    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:34.889629 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:34.889243    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:34.890540 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:34.890510    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:24:34.917319 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:34.917278    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:34.917481 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:34.917354    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:24:34.918810 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:34.918786    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:24:44.889271 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:44.889214    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:24:44.901497 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:44.901466    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:24:44.918094 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:44.918061    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:24:54.889573 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:54.889410    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:24:54.917906 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:24:54.917862    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:25:04.889014 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:04.888962    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:25:04.917842 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:04.917799    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:25:14.888729 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:14.888671    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:25:14.917582 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:14.917544    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:25:16.289040 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:16.289002    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"]
Jul 09 10:25:24.889403 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:24.889357    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:25:24.918143 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:24.918097    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:25:31.312829 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:31.312754    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main" containerID="cri-o://39f903d82d68130fd028634a2954acbc602f8283506f7820eeb7e8f33f68ecce" gracePeriod=45
Jul 09 10:25:31.417901 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:31.417839    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 09 10:25:31.953931 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:31.953868    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerID="39f903d82d68130fd028634a2954acbc602f8283506f7820eeb7e8f33f68ecce" exitCode=0
Jul 09 10:25:31.954138 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:31.953934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" event={"ID":"cd242324-7a6e-4633-82f3-190a80d1cdb9","Type":"ContainerDied","Data":"39f903d82d68130fd028634a2954acbc602f8283506f7820eeb7e8f33f68ecce"}
Jul 09 10:25:31.988268 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:31.988243    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:25:32.061287 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.061190    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqsdm\" (UniqueName: \"kubernetes.io/projected/cd242324-7a6e-4633-82f3-190a80d1cdb9-kube-api-access-cqsdm\") pod \"cd242324-7a6e-4633-82f3-190a80d1cdb9\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") "
Jul 09 10:25:32.061287 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.061255    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-home\") pod \"cd242324-7a6e-4633-82f3-190a80d1cdb9\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") "
Jul 09 10:25:32.061287 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.061281    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-dshm\") pod \"cd242324-7a6e-4633-82f3-190a80d1cdb9\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") "
Jul 09 10:25:32.061567 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.061309    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-tmp-dir\") pod \"cd242324-7a6e-4633-82f3-190a80d1cdb9\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") "
Jul 09 10:25:32.061567 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.061336    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd242324-7a6e-4633-82f3-190a80d1cdb9-tls-certs\") pod \"cd242324-7a6e-4633-82f3-190a80d1cdb9\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") "
Jul 09 10:25:32.061567 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.061367    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-model-cache\") pod \"cd242324-7a6e-4633-82f3-190a80d1cdb9\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") "
Jul 09 10:25:32.061567 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.061393    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-kserve-provision-location\") pod \"cd242324-7a6e-4633-82f3-190a80d1cdb9\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") "
Jul 09 10:25:32.061567 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.061421    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd242324-7a6e-4633-82f3-190a80d1cdb9-cabundle-cert\") pod \"cd242324-7a6e-4633-82f3-190a80d1cdb9\" (UID: \"cd242324-7a6e-4633-82f3-190a80d1cdb9\") "
Jul 09 10:25:32.061815 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.061626    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-home" (OuterVolumeSpecName: "home") pod "cd242324-7a6e-4633-82f3-190a80d1cdb9" (UID: "cd242324-7a6e-4633-82f3-190a80d1cdb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:25:32.061815 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.061778    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd242324-7a6e-4633-82f3-190a80d1cdb9" (UID: "cd242324-7a6e-4633-82f3-190a80d1cdb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:25:32.062090 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.062049    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd242324-7a6e-4633-82f3-190a80d1cdb9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd242324-7a6e-4633-82f3-190a80d1cdb9" (UID: "cd242324-7a6e-4633-82f3-190a80d1cdb9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:25:32.064226 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.064201    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-dshm" (OuterVolumeSpecName: "dshm") pod "cd242324-7a6e-4633-82f3-190a80d1cdb9" (UID: "cd242324-7a6e-4633-82f3-190a80d1cdb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:25:32.064346 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.064207    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd242324-7a6e-4633-82f3-190a80d1cdb9-kube-api-access-cqsdm" (OuterVolumeSpecName: "kube-api-access-cqsdm") pod "cd242324-7a6e-4633-82f3-190a80d1cdb9" (UID: "cd242324-7a6e-4633-82f3-190a80d1cdb9"). InnerVolumeSpecName "kube-api-access-cqsdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:25:32.064346 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.064199    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd242324-7a6e-4633-82f3-190a80d1cdb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd242324-7a6e-4633-82f3-190a80d1cdb9" (UID: "cd242324-7a6e-4633-82f3-190a80d1cdb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:25:32.073822 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.073786    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd242324-7a6e-4633-82f3-190a80d1cdb9" (UID: "cd242324-7a6e-4633-82f3-190a80d1cdb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:25:32.126601 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.126557    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd242324-7a6e-4633-82f3-190a80d1cdb9" (UID: "cd242324-7a6e-4633-82f3-190a80d1cdb9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:25:32.162608 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.162574    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:25:32.162608 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.162607    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:25:32.162795 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.162618    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd242324-7a6e-4633-82f3-190a80d1cdb9-cabundle-cert\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:25:32.162795 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.162627    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqsdm\" (UniqueName: \"kubernetes.io/projected/cd242324-7a6e-4633-82f3-190a80d1cdb9-kube-api-access-cqsdm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:25:32.162795 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.162636    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:25:32.162795 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.162645    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:25:32.162795 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.162670    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd242324-7a6e-4633-82f3-190a80d1cdb9-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:25:32.162795 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.162679    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd242324-7a6e-4633-82f3-190a80d1cdb9-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:25:32.960833 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.960796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv" event={"ID":"cd242324-7a6e-4633-82f3-190a80d1cdb9","Type":"ContainerDied","Data":"7c7ac05fdc13f463436fac8ab1391a5399ec7ac7696dc90da31bfbc8cd24d7b4"}
Jul 09 10:25:32.960833 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.960830    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"
Jul 09 10:25:32.961411 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.960867    2581 scope.go:117] "RemoveContainer" containerID="39f903d82d68130fd028634a2954acbc602f8283506f7820eeb7e8f33f68ecce"
Jul 09 10:25:32.972142 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.972117    2581 scope.go:117] "RemoveContainer" containerID="7677b6633e43ff481b0b9aaf851101ecc3fd33b5f2bca5495c62f4447b77f1d4"
Jul 09 10:25:32.986941 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.986872    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"]
Jul 09 10:25:32.992178 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:32.992155    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-64b48975c9-spbbv"]
Jul 09 10:25:34.228614 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:34.228577    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" path="/var/lib/kubelet/pods/cd242324-7a6e-4633-82f3-190a80d1cdb9/volumes"
Jul 09 10:25:34.888823 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:34.888783    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:25:34.917720 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:34.917680    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:25:40.511103 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.511068    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"]
Jul 09 10:25:40.511516 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.511494    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="storage-initializer"
Jul 09 10:25:40.511561 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.511522    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="storage-initializer"
Jul 09 10:25:40.511561 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.511545    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main"
Jul 09 10:25:40.511561 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.511554    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main"
Jul 09 10:25:40.511673 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.511648    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd242324-7a6e-4633-82f3-190a80d1cdb9" containerName="main"
Jul 09 10:25:40.513869 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.513849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.516307 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.516279    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 09 10:25:40.516428 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.516343    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 09 10:25:40.528731 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.528711    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"]
Jul 09 10:25:40.640615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.640580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1df23051-6ef3-4d45-b5dd-6261ec752a8a-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.640615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.640625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84kxh\" (UniqueName: \"kubernetes.io/projected/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kube-api-access-84kxh\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.640879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.640652    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.640879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.640718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.640879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.640789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-model-cache\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.640879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.640876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-dshm\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.641101 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.640943    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.641101 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.640977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-home\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.741606 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.741563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.741797 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.741630    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.741797 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.741662    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-model-cache\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.741797 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.741744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-dshm\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.741797 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.741780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.742049 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.741814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-home\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.742049 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.741856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1df23051-6ef3-4d45-b5dd-6261ec752a8a-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.742049 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.741884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84kxh\" (UniqueName: \"kubernetes.io/projected/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kube-api-access-84kxh\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.742209 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.742146    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.742266 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.742218    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.742335 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.742311    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-home\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.742335 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.742320    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-model-cache\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.742950 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.742923    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1df23051-6ef3-4d45-b5dd-6261ec752a8a-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.744254 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.744227    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-dshm\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.744549 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.744524    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tls-certs\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.753478 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.753450    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84kxh\" (UniqueName: \"kubernetes.io/projected/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kube-api-access-84kxh\") pod \"lora-multiple-adapters-test-kserve-6f544c69b-vjtx4\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.824824 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.824745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:40.999950 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:40.999920    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"]
Jul 09 10:25:41.004082 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:25:41.004045    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df23051_6ef3_4d45_b5dd_6261ec752a8a.slice/crio-0fe8a2e5e05aa579149734ea281efeaddd24872e99bb650a37a133d25bec8f80 WatchSource:0}: Error finding container 0fe8a2e5e05aa579149734ea281efeaddd24872e99bb650a37a133d25bec8f80: Status 404 returned error can't find the container with id 0fe8a2e5e05aa579149734ea281efeaddd24872e99bb650a37a133d25bec8f80
Jul 09 10:25:42.005383 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:42.005339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" event={"ID":"1df23051-6ef3-4d45-b5dd-6261ec752a8a","Type":"ContainerStarted","Data":"02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0"}
Jul 09 10:25:42.006245 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:42.005461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" event={"ID":"1df23051-6ef3-4d45-b5dd-6261ec752a8a","Type":"ContainerStarted","Data":"0fe8a2e5e05aa579149734ea281efeaddd24872e99bb650a37a133d25bec8f80"}
Jul 09 10:25:44.889450 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:44.889397    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:25:44.917638 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:44.917602    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:25:46.028519 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:46.028484    2581 generic.go:356] "Generic (PLEG): container finished" podID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerID="02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0" exitCode=0
Jul 09 10:25:46.029025 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:46.028549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" event={"ID":"1df23051-6ef3-4d45-b5dd-6261ec752a8a","Type":"ContainerDied","Data":"02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0"}
Jul 09 10:25:47.037503 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:47.037470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" event={"ID":"1df23051-6ef3-4d45-b5dd-6261ec752a8a","Type":"ContainerStarted","Data":"9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f"}
Jul 09 10:25:47.066674 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:47.066619    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" podStartSLOduration=2.656010361 podStartE2EDuration="7.066601933s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.410591572s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:25:47.064563058 +0000 UTC m=+3139.389525378" watchObservedRunningTime="2026-07-09 10:25:47.066601933 +0000 UTC m=+3139.391564218"
Jul 09 10:25:47.825151 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:47.825118    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:50.825395 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:50.825294    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:25:50.827303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:50.827273    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 10:25:54.888603 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:54.888559    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:25:54.917534 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:25:54.917495    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:26:00.825900 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:00.825854    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 10:26:04.888907 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:04.888839    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:26:04.918217 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:04.918176    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:26:10.825779 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:10.825733    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 10:26:14.888860 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:14.888796    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:26:14.918212 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:14.918171    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:26:20.825372 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:20.825331    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 10:26:24.888635 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:24.888583    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:26:24.917688 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:24.917641    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:26:30.825908 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:30.825851    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 10:26:34.888566 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:34.888525    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:26:34.917874 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:34.917826    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:26:40.825840 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:40.825785    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 10:26:44.888926 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:44.888857    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:26:44.918164 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:44.918126    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:26:50.825811 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:50.825767    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 10:26:54.889264 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:54.889219    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:26:54.918012 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:26:54.917972    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:27:00.825263 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:00.825209    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": dial tcp 10.133.0.54:8000: connect: connection refused"
Jul 09 10:27:04.888792 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:04.888746    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8001/health\": dial tcp 10.133.0.52:8001: connect: connection refused"
Jul 09 10:27:04.917835 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:04.917793    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 09 10:27:10.835993 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:10.835955    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:27:10.844215 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:10.844191    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:27:14.904719 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:14.904686    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:27:14.917145 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:14.917123    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:27:14.932590 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:14.932564    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:27:14.942315 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:14.942284    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:27:30.980693 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:30.980600    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"]
Jul 09 10:27:30.984119 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:30.984081    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"]
Jul 09 10:27:46.006814 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:46.006756    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main" containerID="cri-o://aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea" gracePeriod=45
Jul 09 10:27:46.007913 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:46.007862    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main" containerID="cri-o://b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a" gracePeriod=45
Jul 09 10:27:47.133112 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.133010    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="llm-d-routing-sidecar" containerID="cri-o://88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89" gracePeriod=44
Jul 09 10:27:47.292735 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.292710    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:27:47.347854 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.346864    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-kserve-provision-location\") pod \"e6953b3c-6947-4c54-b313-8a4884d866dd\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") "
Jul 09 10:27:47.347854 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.346942    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qpzh\" (UniqueName: \"kubernetes.io/projected/e6953b3c-6947-4c54-b313-8a4884d866dd-kube-api-access-8qpzh\") pod \"e6953b3c-6947-4c54-b313-8a4884d866dd\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") "
Jul 09 10:27:47.347854 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.347010    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-tmp-dir\") pod \"e6953b3c-6947-4c54-b313-8a4884d866dd\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") "
Jul 09 10:27:47.347854 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.347042    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6953b3c-6947-4c54-b313-8a4884d866dd-tls-certs\") pod \"e6953b3c-6947-4c54-b313-8a4884d866dd\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") "
Jul 09 10:27:47.347854 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.347106    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-model-cache\") pod \"e6953b3c-6947-4c54-b313-8a4884d866dd\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") "
Jul 09 10:27:47.347854 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.347140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-dshm\") pod \"e6953b3c-6947-4c54-b313-8a4884d866dd\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") "
Jul 09 10:27:47.347854 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.347170    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-home\") pod \"e6953b3c-6947-4c54-b313-8a4884d866dd\" (UID: \"e6953b3c-6947-4c54-b313-8a4884d866dd\") "
Jul 09 10:27:47.350704 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.350670    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-home" (OuterVolumeSpecName: "home") pod "e6953b3c-6947-4c54-b313-8a4884d866dd" (UID: "e6953b3c-6947-4c54-b313-8a4884d866dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:27:47.351047 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.351024    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6953b3c-6947-4c54-b313-8a4884d866dd" (UID: "e6953b3c-6947-4c54-b313-8a4884d866dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:27:47.355344 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.355317    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6953b3c-6947-4c54-b313-8a4884d866dd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6953b3c-6947-4c54-b313-8a4884d866dd" (UID: "e6953b3c-6947-4c54-b313-8a4884d866dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:27:47.355761 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.355716    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-dshm" (OuterVolumeSpecName: "dshm") pod "e6953b3c-6947-4c54-b313-8a4884d866dd" (UID: "e6953b3c-6947-4c54-b313-8a4884d866dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:27:47.356196 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.356168    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6953b3c-6947-4c54-b313-8a4884d866dd-kube-api-access-8qpzh" (OuterVolumeSpecName: "kube-api-access-8qpzh") pod "e6953b3c-6947-4c54-b313-8a4884d866dd" (UID: "e6953b3c-6947-4c54-b313-8a4884d866dd"). InnerVolumeSpecName "kube-api-access-8qpzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:27:47.372870 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.372840    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6953b3c-6947-4c54-b313-8a4884d866dd" (UID: "e6953b3c-6947-4c54-b313-8a4884d866dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:27:47.387641 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.387612    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:27:47.448580 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.448519    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pwpc\" (UniqueName: \"kubernetes.io/projected/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kube-api-access-7pwpc\") pod \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") "
Jul 09 10:27:47.448781 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.448614    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-home\") pod \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") "
Jul 09 10:27:47.448781 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.448667    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tls-certs\") pod \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") "
Jul 09 10:27:47.448781 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.448756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-model-cache\") pod \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") "
Jul 09 10:27:47.449002 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.448825    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-dshm\") pod \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") "
Jul 09 10:27:47.449002 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.448858    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kserve-provision-location\") pod \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") "
Jul 09 10:27:47.449002 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.448979    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tmp-dir\") pod \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\" (UID: \"a214d4f3-aff1-41f9-8f68-0ee9658b9485\") "
Jul 09 10:27:47.449369 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.449327    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.452879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.452552    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.452879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.450295    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-model-cache" (OuterVolumeSpecName: "model-cache") pod "a214d4f3-aff1-41f9-8f68-0ee9658b9485" (UID: "a214d4f3-aff1-41f9-8f68-0ee9658b9485"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:27:47.452879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.450658    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-home" (OuterVolumeSpecName: "home") pod "a214d4f3-aff1-41f9-8f68-0ee9658b9485" (UID: "a214d4f3-aff1-41f9-8f68-0ee9658b9485"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:27:47.452879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.451999    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a214d4f3-aff1-41f9-8f68-0ee9658b9485" (UID: "a214d4f3-aff1-41f9-8f68-0ee9658b9485"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:27:47.452879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.452572    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.452879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.452634    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qpzh\" (UniqueName: \"kubernetes.io/projected/e6953b3c-6947-4c54-b313-8a4884d866dd-kube-api-access-8qpzh\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.452879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.452672    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.452879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.452688    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6953b3c-6947-4c54-b313-8a4884d866dd-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.453517 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.453013    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kube-api-access-7pwpc" (OuterVolumeSpecName: "kube-api-access-7pwpc") pod "a214d4f3-aff1-41f9-8f68-0ee9658b9485" (UID: "a214d4f3-aff1-41f9-8f68-0ee9658b9485"). InnerVolumeSpecName "kube-api-access-7pwpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:27:47.456295 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.456263    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e6953b3c-6947-4c54-b313-8a4884d866dd" (UID: "e6953b3c-6947-4c54-b313-8a4884d866dd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:27:47.459021 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.458974    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-dshm" (OuterVolumeSpecName: "dshm") pod "a214d4f3-aff1-41f9-8f68-0ee9658b9485" (UID: "a214d4f3-aff1-41f9-8f68-0ee9658b9485"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:27:47.469143 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.469123    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a214d4f3-aff1-41f9-8f68-0ee9658b9485" (UID: "a214d4f3-aff1-41f9-8f68-0ee9658b9485"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:27:47.536426 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.536305    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a214d4f3-aff1-41f9-8f68-0ee9658b9485" (UID: "a214d4f3-aff1-41f9-8f68-0ee9658b9485"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:27:47.553388 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.553340    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pwpc\" (UniqueName: \"kubernetes.io/projected/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kube-api-access-7pwpc\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.553388 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.553379    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.553388 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.553395    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.553764 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.553410    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e6953b3c-6947-4c54-b313-8a4884d866dd-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.553764 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.553425    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.553764 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.553439    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.553764 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.553454    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.553764 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.553470    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a214d4f3-aff1-41f9-8f68-0ee9658b9485-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:27:47.572390 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.572360    2581 generic.go:356] "Generic (PLEG): container finished" podID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerID="aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea" exitCode=0
Jul 09 10:27:47.572538 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.572439    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"
Jul 09 10:27:47.572538 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.572437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" event={"ID":"e6953b3c-6947-4c54-b313-8a4884d866dd","Type":"ContainerDied","Data":"aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea"}
Jul 09 10:27:47.572538 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.572526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g" event={"ID":"e6953b3c-6947-4c54-b313-8a4884d866dd","Type":"ContainerDied","Data":"44ad13dd3413345c7c80a7b0d068b8ed7e72280b5f0ca533b0775e8d6109946e"}
Jul 09 10:27:47.572700 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.572557    2581 scope.go:117] "RemoveContainer" containerID="aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea"
Jul 09 10:27:47.574572 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.574550    2581 generic.go:356] "Generic (PLEG): container finished" podID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerID="b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a" exitCode=0
Jul 09 10:27:47.574572 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.574570    2581 generic.go:356] "Generic (PLEG): container finished" podID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerID="88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89" exitCode=0
Jul 09 10:27:47.574737 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.574617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" event={"ID":"a214d4f3-aff1-41f9-8f68-0ee9658b9485","Type":"ContainerDied","Data":"b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a"}
Jul 09 10:27:47.574737 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.574625    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"
Jul 09 10:27:47.574737 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.574658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" event={"ID":"a214d4f3-aff1-41f9-8f68-0ee9658b9485","Type":"ContainerDied","Data":"88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89"}
Jul 09 10:27:47.574737 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.574684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk" event={"ID":"a214d4f3-aff1-41f9-8f68-0ee9658b9485","Type":"ContainerDied","Data":"41e2a73f3ee8b6891c48bea8e04db3dda55c8fc3cb23dc046ff2d957027bc5bf"}
Jul 09 10:27:47.581744 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.581728    2581 scope.go:117] "RemoveContainer" containerID="bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a"
Jul 09 10:27:47.589898 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.589868    2581 scope.go:117] "RemoveContainer" containerID="aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea"
Jul 09 10:27:47.590138 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:27:47.590120    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea\": container with ID starting with aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea not found: ID does not exist" containerID="aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea"
Jul 09 10:27:47.590192 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.590147    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea"} err="failed to get container status \"aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea\": rpc error: code = NotFound desc = could not find container \"aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea\": container with ID starting with aa7ee785ff5ff28672607f422a0679e70f450b0c5302242931d9bc2976d981ea not found: ID does not exist"
Jul 09 10:27:47.590192 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.590165    2581 scope.go:117] "RemoveContainer" containerID="bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a"
Jul 09 10:27:47.590375 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:27:47.590358    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a\": container with ID starting with bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a not found: ID does not exist" containerID="bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a"
Jul 09 10:27:47.590439 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.590385    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a"} err="failed to get container status \"bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a\": rpc error: code = NotFound desc = could not find container \"bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a\": container with ID starting with bd76c63565de9cf8b802685e9b348ca5ff2a6a096c0ef40d371f530de1ab318a not found: ID does not exist"
Jul 09 10:27:47.590439 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.590406    2581 scope.go:117] "RemoveContainer" containerID="b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a"
Jul 09 10:27:47.595478 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.595456    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"]
Jul 09 10:27:47.598005 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.597992    2581 scope.go:117] "RemoveContainer" containerID="6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3"
Jul 09 10:27:47.601305 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.601286    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-79f7f4c6d6-8bz5g"]
Jul 09 10:27:47.605816 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.605800    2581 scope.go:117] "RemoveContainer" containerID="88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89"
Jul 09 10:27:47.612932 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.612862    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"]
Jul 09 10:27:47.615058 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.615026    2581 scope.go:117] "RemoveContainer" containerID="b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a"
Jul 09 10:27:47.615420 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:27:47.615384    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a\": container with ID starting with b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a not found: ID does not exist" containerID="b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a"
Jul 09 10:27:47.615535 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.615429    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a"} err="failed to get container status \"b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a\": rpc error: code = NotFound desc = could not find container \"b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a\": container with ID starting with b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a not found: ID does not exist"
Jul 09 10:27:47.615535 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.615455    2581 scope.go:117] "RemoveContainer" containerID="6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3"
Jul 09 10:27:47.615753 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:27:47.615731    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3\": container with ID starting with 6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3 not found: ID does not exist" containerID="6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3"
Jul 09 10:27:47.615825 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.615759    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3"} err="failed to get container status \"6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3\": rpc error: code = NotFound desc = could not find container \"6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3\": container with ID starting with 6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3 not found: ID does not exist"
Jul 09 10:27:47.615825 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.615779    2581 scope.go:117] "RemoveContainer" containerID="88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89"
Jul 09 10:27:47.617445 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:27:47.617417    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89\": container with ID starting with 88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89 not found: ID does not exist" containerID="88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89"
Jul 09 10:27:47.617611 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.617587    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89"} err="failed to get container status \"88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89\": rpc error: code = NotFound desc = could not find container \"88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89\": container with ID starting with 88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89 not found: ID does not exist"
Jul 09 10:27:47.617739 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.617713    2581 scope.go:117] "RemoveContainer" containerID="b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a"
Jul 09 10:27:47.618335 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.618305    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a"} err="failed to get container status \"b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a\": rpc error: code = NotFound desc = could not find container \"b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a\": container with ID starting with b87886e95953b06a06a545b1fa1a14d61d71938d0beced828e937766184a334a not found: ID does not exist"
Jul 09 10:27:47.618449 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.618335    2581 scope.go:117] "RemoveContainer" containerID="6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3"
Jul 09 10:27:47.618841 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.618804    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3"} err="failed to get container status \"6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3\": rpc error: code = NotFound desc = could not find container \"6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3\": container with ID starting with 6cee82677b41dca80ed6be5d5faca42977fd67168d38ebb72e58c7044d3a34e3 not found: ID does not exist"
Jul 09 10:27:47.618996 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.618842    2581 scope.go:117] "RemoveContainer" containerID="88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89"
Jul 09 10:27:47.619305 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.619210    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89"} err="failed to get container status \"88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89\": rpc error: code = NotFound desc = could not find container \"88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89\": container with ID starting with 88e076c404a753395e606f82f263fb987503df0e7043af17c6377a4f1d61fc89 not found: ID does not exist"
Jul 09 10:27:47.619425 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:47.619346    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5d7f47bbd7-m2svk"]
Jul 09 10:27:48.232185 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:48.232138    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" path="/var/lib/kubelet/pods/a214d4f3-aff1-41f9-8f68-0ee9658b9485/volumes"
Jul 09 10:27:48.232948 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:27:48.232923    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" path="/var/lib/kubelet/pods/e6953b3c-6947-4c54-b313-8a4884d866dd/volumes"
Jul 09 10:28:04.688123 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688090    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"]
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688365    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="storage-initializer"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688376    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="storage-initializer"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688384    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688389    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688396    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688401    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688415    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="llm-d-routing-sidecar"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688419    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="llm-d-routing-sidecar"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688430    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="storage-initializer"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688435    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="storage-initializer"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688478    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="main"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688483    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a214d4f3-aff1-41f9-8f68-0ee9658b9485" containerName="llm-d-routing-sidecar"
Jul 09 10:28:04.688489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.688489    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6953b3c-6947-4c54-b313-8a4884d866dd" containerName="main"
Jul 09 10:28:04.692222 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.692203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.694566 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.694542    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 09 10:28:04.704739 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.704719    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"]
Jul 09 10:28:04.805460 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.805423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8lnm\" (UniqueName: \"kubernetes.io/projected/446f9530-8fa9-4472-997a-5ff532013056-kube-api-access-p8lnm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.805460 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.805461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.805677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.805484    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446f9530-8fa9-4472-997a-5ff532013056-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.805677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.805555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.805677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.805610    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.805677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.805632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.805677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.805657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.906984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8lnm\" (UniqueName: \"kubernetes.io/projected/446f9530-8fa9-4472-997a-5ff532013056-kube-api-access-p8lnm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.907020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907284 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.907038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446f9530-8fa9-4472-997a-5ff532013056-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907284 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.907160    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907284 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.907227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907284 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.907256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907284 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.907278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907523 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.907372    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907523 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.907485    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907594 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.907572    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.907631 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.907587    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.909502 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.909479    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.909580 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.909546    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446f9530-8fa9-4472-997a-5ff532013056-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:04.915543 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:04.915521    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8lnm\" (UniqueName: \"kubernetes.io/projected/446f9530-8fa9-4472-997a-5ff532013056-kube-api-access-p8lnm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:05.001117 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:05.001088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:05.131221 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:05.131191    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"]
Jul 09 10:28:05.133707 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:28:05.133683    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446f9530_8fa9_4472_997a_5ff532013056.slice/crio-03cf9efaada13f1ae7f5b7dc46be5c482d6eca1695ec23967989e9d512d5dc57 WatchSource:0}: Error finding container 03cf9efaada13f1ae7f5b7dc46be5c482d6eca1695ec23967989e9d512d5dc57: Status 404 returned error can't find the container with id 03cf9efaada13f1ae7f5b7dc46be5c482d6eca1695ec23967989e9d512d5dc57
Jul 09 10:28:05.668334 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:05.668292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" event={"ID":"446f9530-8fa9-4472-997a-5ff532013056","Type":"ContainerStarted","Data":"6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9"}
Jul 09 10:28:05.668526 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:05.668362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" event={"ID":"446f9530-8fa9-4472-997a-5ff532013056","Type":"ContainerStarted","Data":"03cf9efaada13f1ae7f5b7dc46be5c482d6eca1695ec23967989e9d512d5dc57"}
Jul 09 10:28:05.775244 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:05.775213    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"]
Jul 09 10:28:08.684984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:08.684882    2581 generic.go:356] "Generic (PLEG): container finished" podID="446f9530-8fa9-4472-997a-5ff532013056" containerID="6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9" exitCode=0
Jul 09 10:28:08.684984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:08.684953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" event={"ID":"446f9530-8fa9-4472-997a-5ff532013056","Type":"ContainerDied","Data":"6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9"}
Jul 09 10:28:09.690747 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:09.690713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" event={"ID":"446f9530-8fa9-4472-997a-5ff532013056","Type":"ContainerStarted","Data":"4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e"}
Jul 09 10:28:09.715994 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:09.715937    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podStartSLOduration=2.576605546 podStartE2EDuration="5.715916743s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.139311197s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:28:09.713862402 +0000 UTC m=+3282.038824688" watchObservedRunningTime="2026-07-09 10:28:09.715916743 +0000 UTC m=+3282.040879024"
Jul 09 10:28:10.002201 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:10.002157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:15.001911 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:15.001856    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:28:15.003205 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:15.003173    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:28:20.801760 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:20.801700    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main" containerID="cri-o://9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f" gracePeriod=45
Jul 09 10:28:21.468156 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.468134    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:28:21.555274 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555190    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tls-certs\") pod \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") "
Jul 09 10:28:21.555274 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555228    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1df23051-6ef3-4d45-b5dd-6261ec752a8a-cabundle-cert\") pod \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") "
Jul 09 10:28:21.555274 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555245    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-dshm\") pod \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") "
Jul 09 10:28:21.555274 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84kxh\" (UniqueName: \"kubernetes.io/projected/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kube-api-access-84kxh\") pod \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") "
Jul 09 10:28:21.555620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555349    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tmp-dir\") pod \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") "
Jul 09 10:28:21.555620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555399    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-model-cache\") pod \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") "
Jul 09 10:28:21.555620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kserve-provision-location\") pod \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") "
Jul 09 10:28:21.555620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-home\") pod \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\" (UID: \"1df23051-6ef3-4d45-b5dd-6261ec752a8a\") "
Jul 09 10:28:21.555620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555607    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df23051-6ef3-4d45-b5dd-6261ec752a8a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1df23051-6ef3-4d45-b5dd-6261ec752a8a" (UID: "1df23051-6ef3-4d45-b5dd-6261ec752a8a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 10:28:21.555884 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555763    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1df23051-6ef3-4d45-b5dd-6261ec752a8a-cabundle-cert\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:28:21.556029 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.555999    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-home" (OuterVolumeSpecName: "home") pod "1df23051-6ef3-4d45-b5dd-6261ec752a8a" (UID: "1df23051-6ef3-4d45-b5dd-6261ec752a8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:28:21.556260 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.556226    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-model-cache" (OuterVolumeSpecName: "model-cache") pod "1df23051-6ef3-4d45-b5dd-6261ec752a8a" (UID: "1df23051-6ef3-4d45-b5dd-6261ec752a8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:28:21.557867 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.557834    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-dshm" (OuterVolumeSpecName: "dshm") pod "1df23051-6ef3-4d45-b5dd-6261ec752a8a" (UID: "1df23051-6ef3-4d45-b5dd-6261ec752a8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:28:21.557980 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.557931    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1df23051-6ef3-4d45-b5dd-6261ec752a8a" (UID: "1df23051-6ef3-4d45-b5dd-6261ec752a8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:28:21.558200 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.558175    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kube-api-access-84kxh" (OuterVolumeSpecName: "kube-api-access-84kxh") pod "1df23051-6ef3-4d45-b5dd-6261ec752a8a" (UID: "1df23051-6ef3-4d45-b5dd-6261ec752a8a"). InnerVolumeSpecName "kube-api-access-84kxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:28:21.569203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.569170    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1df23051-6ef3-4d45-b5dd-6261ec752a8a" (UID: "1df23051-6ef3-4d45-b5dd-6261ec752a8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:28:21.618423 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.618381    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1df23051-6ef3-4d45-b5dd-6261ec752a8a" (UID: "1df23051-6ef3-4d45-b5dd-6261ec752a8a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:28:21.656368 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.656338    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:28:21.656368 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.656362    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:28:21.656368 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.656371    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:28:21.656579 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.656380    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:28:21.656579 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.656388    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84kxh\" (UniqueName: \"kubernetes.io/projected/1df23051-6ef3-4d45-b5dd-6261ec752a8a-kube-api-access-84kxh\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:28:21.656579 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.656395    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:28:21.656579 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.656403    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1df23051-6ef3-4d45-b5dd-6261ec752a8a-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:28:21.744750 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.744719    2581 generic.go:356] "Generic (PLEG): container finished" podID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerID="9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f" exitCode=0
Jul 09 10:28:21.744921 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.744762    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"
Jul 09 10:28:21.744921 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.744762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" event={"ID":"1df23051-6ef3-4d45-b5dd-6261ec752a8a","Type":"ContainerDied","Data":"9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f"}
Jul 09 10:28:21.744921 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.744914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4" event={"ID":"1df23051-6ef3-4d45-b5dd-6261ec752a8a","Type":"ContainerDied","Data":"0fe8a2e5e05aa579149734ea281efeaddd24872e99bb650a37a133d25bec8f80"}
Jul 09 10:28:21.745096 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.744950    2581 scope.go:117] "RemoveContainer" containerID="9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f"
Jul 09 10:28:21.757638 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.757614    2581 scope.go:117] "RemoveContainer" containerID="02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0"
Jul 09 10:28:21.765721 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.765700    2581 scope.go:117] "RemoveContainer" containerID="9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f"
Jul 09 10:28:21.765983 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:28:21.765962    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f\": container with ID starting with 9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f not found: ID does not exist" containerID="9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f"
Jul 09 10:28:21.766036 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.765991    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f"} err="failed to get container status \"9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f\": rpc error: code = NotFound desc = could not find container \"9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f\": container with ID starting with 9feea73f11b00d82f71cfdb384328142a54ae97e861da16402d995c4c22d1d7f not found: ID does not exist"
Jul 09 10:28:21.766036 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.766009    2581 scope.go:117] "RemoveContainer" containerID="02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0"
Jul 09 10:28:21.766235 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:28:21.766217    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0\": container with ID starting with 02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0 not found: ID does not exist" containerID="02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0"
Jul 09 10:28:21.766275 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.766239    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0"} err="failed to get container status \"02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0\": rpc error: code = NotFound desc = could not find container \"02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0\": container with ID starting with 02a726187a473603e8780bcf2e9b1b21a26e7a6f43ab7287c3d53ace6c1067a0 not found: ID does not exist"
Jul 09 10:28:21.772150 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.772128    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"]
Jul 09 10:28:21.777844 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:21.777822    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-6f544c69b-vjtx4"]
Jul 09 10:28:22.227743 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:22.227712    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" path="/var/lib/kubelet/pods/1df23051-6ef3-4d45-b5dd-6261ec752a8a/volumes"
Jul 09 10:28:25.001683 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:25.001640    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:28:30.188935 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:30.188908    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:28:30.195905 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:30.195862    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:28:33.998773 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:33.998735    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"]
Jul 09 10:28:33.999204 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:33.999118    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main"
Jul 09 10:28:33.999204 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:33.999137    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main"
Jul 09 10:28:33.999204 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:33.999152    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="storage-initializer"
Jul 09 10:28:33.999204 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:33.999160    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="storage-initializer"
Jul 09 10:28:33.999353 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:33.999230    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1df23051-6ef3-4d45-b5dd-6261ec752a8a" containerName="main"
Jul 09 10:28:34.002347 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.002317    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.004599 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.004580    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 09 10:28:34.021910 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.015470    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"]
Jul 09 10:28:34.047807 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.047781    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tls-certs\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.047807 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.047810    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tmp-dir\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.048020 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.047839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cw87\" (UniqueName: \"kubernetes.io/projected/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kube-api-access-4cw87\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.048020 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.047919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kserve-provision-location\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.048020 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.048011    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-home\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.048148 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.048036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-dshm\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.048148 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.048070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-model-cache\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.149165 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.149128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tls-certs\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.149165 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.149166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tmp-dir\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.149423 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.149197    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cw87\" (UniqueName: \"kubernetes.io/projected/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kube-api-access-4cw87\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.149423 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.149216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kserve-provision-location\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.149423 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.149261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-home\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.149423 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.149275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-dshm\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.149423 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.149293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-model-cache\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.149695 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.149657    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tmp-dir\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.149747 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.149705    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-home\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.149747 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.149678    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-model-cache\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.150041 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.150019    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kserve-provision-location\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.151568 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.151544    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-dshm\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.151834 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.151814    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tls-certs\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.157921 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.157878    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cw87\" (UniqueName: \"kubernetes.io/projected/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kube-api-access-4cw87\") pod \"tls-verification-test-kserve-c5d84c7bb-8mkg9\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.316950 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.316870    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:34.449851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.449821    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"]
Jul 09 10:28:34.451699 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:28:34.451671    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad773c9e_e408_456f_ae0b_9f5ed94a7cc1.slice/crio-e03e3b654b49bbc9b579c5979d8f959bacc601189fc650decfb3c9109b2041ac WatchSource:0}: Error finding container e03e3b654b49bbc9b579c5979d8f959bacc601189fc650decfb3c9109b2041ac: Status 404 returned error can't find the container with id e03e3b654b49bbc9b579c5979d8f959bacc601189fc650decfb3c9109b2041ac
Jul 09 10:28:34.800482 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.800435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" event={"ID":"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1","Type":"ContainerStarted","Data":"18fcc4ff91d10be76303f6e423b912c48d3622c62f0133ff2846ffb9e3cf654b"}
Jul 09 10:28:34.800685 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:34.800495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" event={"ID":"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1","Type":"ContainerStarted","Data":"e03e3b654b49bbc9b579c5979d8f959bacc601189fc650decfb3c9109b2041ac"}
Jul 09 10:28:35.001816 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:35.001760    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:28:38.819837 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:38.819802    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerID="18fcc4ff91d10be76303f6e423b912c48d3622c62f0133ff2846ffb9e3cf654b" exitCode=0
Jul 09 10:28:38.820239 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:38.819859    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" event={"ID":"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1","Type":"ContainerDied","Data":"18fcc4ff91d10be76303f6e423b912c48d3622c62f0133ff2846ffb9e3cf654b"}
Jul 09 10:28:39.825652 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:39.825617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" event={"ID":"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1","Type":"ContainerStarted","Data":"bdb4c0997ebbf6ab4a8a49745e7528cba47985d4677343ea20da2d2e10fc7725"}
Jul 09 10:28:39.857836 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:39.857775    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podStartSLOduration=3.476694362 podStartE2EDuration="6.857756901s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.381062539s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:28:39.857212971 +0000 UTC m=+3312.182175257" watchObservedRunningTime="2026-07-09 10:28:39.857756901 +0000 UTC m=+3312.182719188"
Jul 09 10:28:40.317521 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:40.317483    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:44.317681 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:44.317647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:28:44.319305 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:44.319263    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:28:45.002186 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:45.002144    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:28:54.318159 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:54.318066    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:28:55.001671 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:28:55.001620    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:29:04.317569 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:04.317529    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:29:05.001845 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:05.001798    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:29:14.317324 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:14.317281    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:29:15.002574 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:15.002529    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:29:24.317456 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:24.317412    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:29:25.001904 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:25.001851    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:29:34.318071 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:34.318028    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:29:35.002571 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:35.002528    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:29:44.317492 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:44.317445    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:29:45.002531 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:45.002481    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:29:54.317721 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:54.317682    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:29:55.002375 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:29:55.002328    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:30:04.318172 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:04.318129    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:30:05.002009 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:05.001962    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:30:14.318126 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:14.318074    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:30:15.011961 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:15.011927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:30:15.020100 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:15.020074    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:30:23.190359 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:23.190280    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"]
Jul 09 10:30:24.317687 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:24.317639    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:30:34.317971 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:34.317930    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:30:38.213103 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:38.213044    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" containerID="cri-o://4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e" gracePeriod=45
Jul 09 10:30:39.013293 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.013257    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 09 10:30:39.196027 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.196005    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:30:39.313906 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.313792    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-tmp-dir\") pod \"446f9530-8fa9-4472-997a-5ff532013056\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") "
Jul 09 10:30:39.313906 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.313836    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-model-cache\") pod \"446f9530-8fa9-4472-997a-5ff532013056\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") "
Jul 09 10:30:39.313906 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.313864    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446f9530-8fa9-4472-997a-5ff532013056-tls-certs\") pod \"446f9530-8fa9-4472-997a-5ff532013056\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") "
Jul 09 10:30:39.314490 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.313932    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8lnm\" (UniqueName: \"kubernetes.io/projected/446f9530-8fa9-4472-997a-5ff532013056-kube-api-access-p8lnm\") pod \"446f9530-8fa9-4472-997a-5ff532013056\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") "
Jul 09 10:30:39.314490 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.313977    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-kserve-provision-location\") pod \"446f9530-8fa9-4472-997a-5ff532013056\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") "
Jul 09 10:30:39.314490 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.314000    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-dshm\") pod \"446f9530-8fa9-4472-997a-5ff532013056\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") "
Jul 09 10:30:39.314490 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.314046    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-home\") pod \"446f9530-8fa9-4472-997a-5ff532013056\" (UID: \"446f9530-8fa9-4472-997a-5ff532013056\") "
Jul 09 10:30:39.314490 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.314162    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-model-cache" (OuterVolumeSpecName: "model-cache") pod "446f9530-8fa9-4472-997a-5ff532013056" (UID: "446f9530-8fa9-4472-997a-5ff532013056"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:30:39.314490 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.314256    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:30:39.314805 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.314679    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-home" (OuterVolumeSpecName: "home") pod "446f9530-8fa9-4472-997a-5ff532013056" (UID: "446f9530-8fa9-4472-997a-5ff532013056"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:30:39.317019 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.316974    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/446f9530-8fa9-4472-997a-5ff532013056-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "446f9530-8fa9-4472-997a-5ff532013056" (UID: "446f9530-8fa9-4472-997a-5ff532013056"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:30:39.317152 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.317105    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446f9530-8fa9-4472-997a-5ff532013056-kube-api-access-p8lnm" (OuterVolumeSpecName: "kube-api-access-p8lnm") pod "446f9530-8fa9-4472-997a-5ff532013056" (UID: "446f9530-8fa9-4472-997a-5ff532013056"). InnerVolumeSpecName "kube-api-access-p8lnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:30:39.324055 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.324013    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-dshm" (OuterVolumeSpecName: "dshm") pod "446f9530-8fa9-4472-997a-5ff532013056" (UID: "446f9530-8fa9-4472-997a-5ff532013056"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:30:39.327123 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.326431    2581 generic.go:356] "Generic (PLEG): container finished" podID="446f9530-8fa9-4472-997a-5ff532013056" containerID="4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e" exitCode=0
Jul 09 10:30:39.327123 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.326471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" event={"ID":"446f9530-8fa9-4472-997a-5ff532013056","Type":"ContainerDied","Data":"4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e"}
Jul 09 10:30:39.327123 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.326521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk" event={"ID":"446f9530-8fa9-4472-997a-5ff532013056","Type":"ContainerDied","Data":"03cf9efaada13f1ae7f5b7dc46be5c482d6eca1695ec23967989e9d512d5dc57"}
Jul 09 10:30:39.327123 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.326551    2581 scope.go:117] "RemoveContainer" containerID="4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e"
Jul 09 10:30:39.327123 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.326717    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"
Jul 09 10:30:39.336037 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.336009    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "446f9530-8fa9-4472-997a-5ff532013056" (UID: "446f9530-8fa9-4472-997a-5ff532013056"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:30:39.342816 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.342796    2581 scope.go:117] "RemoveContainer" containerID="6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9"
Jul 09 10:30:39.355526 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.355382    2581 scope.go:117] "RemoveContainer" containerID="4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e"
Jul 09 10:30:39.355826 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:30:39.355786    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e\": container with ID starting with 4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e not found: ID does not exist" containerID="4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e"
Jul 09 10:30:39.355967 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.355830    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e"} err="failed to get container status \"4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e\": rpc error: code = NotFound desc = could not find container \"4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e\": container with ID starting with 4bf6f272888900f518c5177e2f430569f119adddf34644595cae931adc2b233e not found: ID does not exist"
Jul 09 10:30:39.355967 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.355860    2581 scope.go:117] "RemoveContainer" containerID="6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9"
Jul 09 10:30:39.356544 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:30:39.356434    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9\": container with ID starting with 6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9 not found: ID does not exist" containerID="6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9"
Jul 09 10:30:39.356544 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.356478    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9"} err="failed to get container status \"6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9\": rpc error: code = NotFound desc = could not find container \"6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9\": container with ID starting with 6dfcf6574e758b38a6026ecd8cc658c9832ea8ca8e4371a9666196a585d603d9 not found: ID does not exist"
Jul 09 10:30:39.404674 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.404631    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "446f9530-8fa9-4472-997a-5ff532013056" (UID: "446f9530-8fa9-4472-997a-5ff532013056"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:30:39.414931 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.414879    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:30:39.414931 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.414930    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:30:39.415160 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.414950    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:30:39.415160 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.414963    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/446f9530-8fa9-4472-997a-5ff532013056-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:30:39.415160 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.414980    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/446f9530-8fa9-4472-997a-5ff532013056-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:30:39.415160 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.414995    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8lnm\" (UniqueName: \"kubernetes.io/projected/446f9530-8fa9-4472-997a-5ff532013056-kube-api-access-p8lnm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:30:39.654095 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.654060    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"]
Jul 09 10:30:39.663164 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:39.663133    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-89679586f-hscsk"]
Jul 09 10:30:40.227923 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:40.227874    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="446f9530-8fa9-4472-997a-5ff532013056" path="/var/lib/kubelet/pods/446f9530-8fa9-4472-997a-5ff532013056/volumes"
Jul 09 10:30:44.327784 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:44.327754    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:30:44.335675 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:44.335651    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:30:53.645577 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.645545    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 10:30:53.646044 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.645833    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="storage-initializer"
Jul 09 10:30:53.646044 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.645845    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="storage-initializer"
Jul 09 10:30:53.646044 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.645864    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main"
Jul 09 10:30:53.646044 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.645869    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main"
Jul 09 10:30:53.646044 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.645934    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="446f9530-8fa9-4472-997a-5ff532013056" containerName="main"
Jul 09 10:30:53.648034 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.648018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.652355 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.652327    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-djs6w\""
Jul 09 10:30:53.652494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.652437    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 09 10:30:53.667072 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.667052    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 10:30:53.721765 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.721731    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 10:30:53.724056 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.724034    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.728114 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.728087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.728211 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.728165    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.728260 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.728223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.728301 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.728264    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.728345 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.728305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhh9\" (UniqueName: \"kubernetes.io/projected/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kube-api-access-fjhh9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.728345 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.728339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.728420 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.728362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.742054 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.742030    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 10:30:53.829464 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829437    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.829464 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.829677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9260a465-e723-4f54-8635-41793ab8f09c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.829677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.829677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.829677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdmt\" (UniqueName: \"kubernetes.io/projected/9260a465-e723-4f54-8635-41793ab8f09c-kube-api-access-qfdmt\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.829677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.829677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.829677 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.829986 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.829986 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhh9\" (UniqueName: \"kubernetes.io/projected/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kube-api-access-fjhh9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.829986 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829841    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.829986 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.829986 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829928    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.830246 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.829993    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.830246 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.830047    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.830246 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.830191    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.830398 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.830264    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.831632 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.831603    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.832095 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.832079    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.852691 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.852662    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhh9\" (UniqueName: \"kubernetes.io/projected/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kube-api-access-fjhh9\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:53.930363 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.930363 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930321    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.930363 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.930620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9260a465-e723-4f54-8635-41793ab8f09c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.930620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.930620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.930620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdmt\" (UniqueName: \"kubernetes.io/projected/9260a465-e723-4f54-8635-41793ab8f09c-kube-api-access-qfdmt\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.930834 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930645    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.930834 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930696    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.930834 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930773    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.930973 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.930831    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.932505 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.932486    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.932796 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.932779    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9260a465-e723-4f54-8635-41793ab8f09c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.938735 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.938711    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdmt\" (UniqueName: \"kubernetes.io/projected/9260a465-e723-4f54-8635-41793ab8f09c-kube-api-access-qfdmt\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:53.957631 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:53.957609    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:30:54.033764 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:54.033724    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:30:54.099167 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:54.099135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 10:30:54.101099 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:30:54.101061    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6fca231_0c40_44e2_80fe_5cb7ce60b07e.slice/crio-af7a26e89e50f4e15217be7682b19077276d3916208ea3ba9ef41c7b18d01ebd WatchSource:0}: Error finding container af7a26e89e50f4e15217be7682b19077276d3916208ea3ba9ef41c7b18d01ebd: Status 404 returned error can't find the container with id af7a26e89e50f4e15217be7682b19077276d3916208ea3ba9ef41c7b18d01ebd
Jul 09 10:30:54.186847 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:54.186786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 10:30:54.189587 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:30:54.189559    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9260a465_e723_4f54_8635_41793ab8f09c.slice/crio-5991174c31f6e03bddea0df1c390ecee6aef256fdd237108fcdac406b21949d4 WatchSource:0}: Error finding container 5991174c31f6e03bddea0df1c390ecee6aef256fdd237108fcdac406b21949d4: Status 404 returned error can't find the container with id 5991174c31f6e03bddea0df1c390ecee6aef256fdd237108fcdac406b21949d4
Jul 09 10:30:54.383150 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:54.383099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"9260a465-e723-4f54-8635-41793ab8f09c","Type":"ContainerStarted","Data":"0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421"}
Jul 09 10:30:54.383366 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:54.383180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"9260a465-e723-4f54-8635-41793ab8f09c","Type":"ContainerStarted","Data":"5991174c31f6e03bddea0df1c390ecee6aef256fdd237108fcdac406b21949d4"}
Jul 09 10:30:54.384827 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:54.384798    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f6fca231-0c40-44e2-80fe-5cb7ce60b07e","Type":"ContainerStarted","Data":"39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15"}
Jul 09 10:30:54.384953 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:54.384852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f6fca231-0c40-44e2-80fe-5cb7ce60b07e","Type":"ContainerStarted","Data":"af7a26e89e50f4e15217be7682b19077276d3916208ea3ba9ef41c7b18d01ebd"}
Jul 09 10:30:58.403441 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:58.403404    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerID="39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15" exitCode=0
Jul 09 10:30:58.403870 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:58.403491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f6fca231-0c40-44e2-80fe-5cb7ce60b07e","Type":"ContainerDied","Data":"39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15"}
Jul 09 10:30:58.405017 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:58.404992    2581 generic.go:356] "Generic (PLEG): container finished" podID="9260a465-e723-4f54-8635-41793ab8f09c" containerID="0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421" exitCode=0
Jul 09 10:30:58.405101 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:58.405063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"9260a465-e723-4f54-8635-41793ab8f09c","Type":"ContainerDied","Data":"0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421"}
Jul 09 10:30:59.412524 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:59.412484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"9260a465-e723-4f54-8635-41793ab8f09c","Type":"ContainerStarted","Data":"d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4"}
Jul 09 10:30:59.414644 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:59.414615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f6fca231-0c40-44e2-80fe-5cb7ce60b07e","Type":"ContainerStarted","Data":"31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666"}
Jul 09 10:30:59.436348 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:59.436292    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.753090077 podStartE2EDuration="6.436276729s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.683186652s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:30:59.434678067 +0000 UTC m=+3451.759640352" watchObservedRunningTime="2026-07-09 10:30:59.436276729 +0000 UTC m=+3451.761239013"
Jul 09 10:30:59.458818 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:59.458752    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.808331227 podStartE2EDuration="6.458736026s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.650404799s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:30:59.458664206 +0000 UTC m=+3451.783626490" watchObservedRunningTime="2026-07-09 10:30:59.458736026 +0000 UTC m=+3451.783698309"
Jul 09 10:30:59.958086 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:30:59.958043    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:31:03.958620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:03.958581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:31:03.960329 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:03.960291    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:31:11.883119 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:11.883078    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"]
Jul 09 10:31:13.958140 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:13.958088    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:31:23.958909 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:23.958847    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:31:26.907179 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:26.907121    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" containerID="cri-o://bdb4c0997ebbf6ab4a8a49745e7528cba47985d4677343ea20da2d2e10fc7725" gracePeriod=45
Jul 09 10:31:27.329642 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.329279    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 09 10:31:27.542880 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.542847    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerID="bdb4c0997ebbf6ab4a8a49745e7528cba47985d4677343ea20da2d2e10fc7725" exitCode=0
Jul 09 10:31:27.543073 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.542996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" event={"ID":"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1","Type":"ContainerDied","Data":"bdb4c0997ebbf6ab4a8a49745e7528cba47985d4677343ea20da2d2e10fc7725"}
Jul 09 10:31:27.667406 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.667382    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:31:27.750761 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.750730    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-model-cache\") pod \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") "
Jul 09 10:31:27.750942 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.750776    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cw87\" (UniqueName: \"kubernetes.io/projected/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kube-api-access-4cw87\") pod \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") "
Jul 09 10:31:27.750942 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.750804    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tmp-dir\") pod \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") "
Jul 09 10:31:27.750942 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.750847    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-dshm\") pod \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") "
Jul 09 10:31:27.750942 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.750925    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tls-certs\") pod \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") "
Jul 09 10:31:27.751163 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.750949    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kserve-provision-location\") pod \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") "
Jul 09 10:31:27.751163 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.750986    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-home\") pod \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\" (UID: \"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1\") "
Jul 09 10:31:27.751163 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.751082    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" (UID: "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:31:27.751322 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.751282    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:31:27.751773 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.751746    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-home" (OuterVolumeSpecName: "home") pod "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" (UID: "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:31:27.753163 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.753131    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kube-api-access-4cw87" (OuterVolumeSpecName: "kube-api-access-4cw87") pod "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" (UID: "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1"). InnerVolumeSpecName "kube-api-access-4cw87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:31:27.753841 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.753817    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" (UID: "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:31:27.753951 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.753882    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-dshm" (OuterVolumeSpecName: "dshm") pod "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" (UID: "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:31:27.769758 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.769719    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" (UID: "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:31:27.821721 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.821682    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" (UID: "ad773c9e-e408-456f-ae0b-9f5ed94a7cc1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:31:27.852460 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.852425    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cw87\" (UniqueName: \"kubernetes.io/projected/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kube-api-access-4cw87\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:31:27.852604 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.852466    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:31:27.852604 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.852482    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:31:27.852604 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.852495    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:31:27.852604 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.852509    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:31:27.852604 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:27.852521    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:31:28.548366 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:28.548344    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"
Jul 09 10:31:28.548739 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:28.548337    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9" event={"ID":"ad773c9e-e408-456f-ae0b-9f5ed94a7cc1","Type":"ContainerDied","Data":"e03e3b654b49bbc9b579c5979d8f959bacc601189fc650decfb3c9109b2041ac"}
Jul 09 10:31:28.548739 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:28.548492    2581 scope.go:117] "RemoveContainer" containerID="bdb4c0997ebbf6ab4a8a49745e7528cba47985d4677343ea20da2d2e10fc7725"
Jul 09 10:31:28.560576 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:28.560555    2581 scope.go:117] "RemoveContainer" containerID="18fcc4ff91d10be76303f6e423b912c48d3622c62f0133ff2846ffb9e3cf654b"
Jul 09 10:31:28.569650 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:28.569630    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"]
Jul 09 10:31:28.575091 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:28.575071    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-c5d84c7bb-8mkg9"]
Jul 09 10:31:30.228392 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:30.228355    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" path="/var/lib/kubelet/pods/ad773c9e-e408-456f-ae0b-9f5ed94a7cc1/volumes"
Jul 09 10:31:33.958391 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:33.958344    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:31:39.627257 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.627219    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"]
Jul 09 10:31:39.627718 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.627679    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main"
Jul 09 10:31:39.627718 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.627697    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main"
Jul 09 10:31:39.627848 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.627725    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="storage-initializer"
Jul 09 10:31:39.627848 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.627736    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="storage-initializer"
Jul 09 10:31:39.627848 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.627823    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad773c9e-e408-456f-ae0b-9f5ed94a7cc1" containerName="main"
Jul 09 10:31:39.631388 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.631363    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.633866 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.633841    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 09 10:31:39.662975 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.662944    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"]
Jul 09 10:31:39.755910 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.755852    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-home\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.756119 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.755919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-kserve-provision-location\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.756119 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.755960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-tmp-dir\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.756119 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.755979    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-dshm\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.756119 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.756019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668b7870-23e5-4385-9d61-30a6461083c5-tls-certs\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.756119 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.756097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cwwr\" (UniqueName: \"kubernetes.io/projected/668b7870-23e5-4385-9d61-30a6461083c5-kube-api-access-6cwwr\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.756421 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.756171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-model-cache\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857164    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-model-cache\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-home\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857478 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-kserve-provision-location\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857478 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-tmp-dir\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857478 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-dshm\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857642 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857476    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668b7870-23e5-4385-9d61-30a6461083c5-tls-certs\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857642 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cwwr\" (UniqueName: \"kubernetes.io/projected/668b7870-23e5-4385-9d61-30a6461083c5-kube-api-access-6cwwr\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857642 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857630    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-home\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857814 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857712    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-kserve-provision-location\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857871 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857855    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-model-cache\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.857961 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.857872    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-tmp-dir\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.859947 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.859921    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-dshm\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.860133 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.860115    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668b7870-23e5-4385-9d61-30a6461083c5-tls-certs\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.865871 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.865844    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cwwr\" (UniqueName: \"kubernetes.io/projected/668b7870-23e5-4385-9d61-30a6461083c5-kube-api-access-6cwwr\") pod \"prestop-hook-test-kserve-789646768b-vjgdj\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:39.944994 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:39.944881    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:40.094497 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:40.094472    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"]
Jul 09 10:31:40.097130 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:31:40.097103    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod668b7870_23e5_4385_9d61_30a6461083c5.slice/crio-34b340b131d67d8fad69638d135c9db7969b8252527744e012437d3d6cc783b7 WatchSource:0}: Error finding container 34b340b131d67d8fad69638d135c9db7969b8252527744e012437d3d6cc783b7: Status 404 returned error can't find the container with id 34b340b131d67d8fad69638d135c9db7969b8252527744e012437d3d6cc783b7
Jul 09 10:31:40.601801 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:40.601763    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" event={"ID":"668b7870-23e5-4385-9d61-30a6461083c5","Type":"ContainerStarted","Data":"d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283"}
Jul 09 10:31:40.601984 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:40.601829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" event={"ID":"668b7870-23e5-4385-9d61-30a6461083c5","Type":"ContainerStarted","Data":"34b340b131d67d8fad69638d135c9db7969b8252527744e012437d3d6cc783b7"}
Jul 09 10:31:43.958366 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:43.958323    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:31:44.622539 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:44.622499    2581 generic.go:356] "Generic (PLEG): container finished" podID="668b7870-23e5-4385-9d61-30a6461083c5" containerID="d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283" exitCode=0
Jul 09 10:31:44.622744 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:44.622580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" event={"ID":"668b7870-23e5-4385-9d61-30a6461083c5","Type":"ContainerDied","Data":"d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283"}
Jul 09 10:31:45.629171 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:45.629129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" event={"ID":"668b7870-23e5-4385-9d61-30a6461083c5","Type":"ContainerStarted","Data":"7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431"}
Jul 09 10:31:45.665685 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:45.665606    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podStartSLOduration=2.93140934 podStartE2EDuration="6.665580862s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.734171522s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:31:45.663832897 +0000 UTC m=+3497.988795181" watchObservedRunningTime="2026-07-09 10:31:45.665580862 +0000 UTC m=+3497.990543150"
Jul 09 10:31:45.945159 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:45.945043    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:49.946078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:49.945983    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:31:49.947741 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:49.947705    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:31:53.958508 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:53.958463    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:31:59.945434 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:31:59.945392    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:32:03.958746 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:03.958700    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:32:09.946165 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:09.946125    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:32:13.958276 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:13.958241    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:32:19.946336 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:19.946293    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:32:23.958692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:23.958644    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:32:29.945739 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:29.945692    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:32:33.958431 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:33.958386    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:32:39.945351 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:39.945301    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:32:43.958425 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:43.958380    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:32:49.946172 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:49.946131    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:32:53.958562 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:53.958501    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:32:59.946249 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:32:59.946200    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:33:03.958842 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:03.958802    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:33:09.946249 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:09.946210    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:33:13.958628 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:13.958593    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:33:19.946013 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:19.945920    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:33:23.973538 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:23.973512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:33:23.985081 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:23.985049    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:33:29.946143 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:29.946098    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:33:30.223778 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:30.223740    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:33:30.229452 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:30.229431    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:33:39.945723 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:39.945679    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:33:49.945691 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:49.945648    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 09 10:33:59.955616 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:59.955587    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:33:59.963412 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:33:59.963385    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:34:03.200428 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.200394    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"]
Jul 09 10:34:03.284506 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.284463    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"]
Jul 09 10:34:03.287482 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.287465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.289919 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:34:03.289899    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"prestop-hook-test-kserve-789646768b-swqvb\" is forbidden: User \"system:node:ip-10-0-129-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-129-157.ec2.internal' and this object" podUID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.313805 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.313779    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"]
Jul 09 10:34:03.387611 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.387581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2ds7\" (UniqueName: \"kubernetes.io/projected/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kube-api-access-c2ds7\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.387611 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.387615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-home\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.387853 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.387647    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tmp-dir\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.387853 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.387668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kserve-provision-location\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.387853 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.387694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-model-cache\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.387853 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.387732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-dshm\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.387853 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.387795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tls-certs\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488191 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2ds7\" (UniqueName: \"kubernetes.io/projected/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kube-api-access-c2ds7\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488386 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-home\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488386 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tmp-dir\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488386 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488297    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kserve-provision-location\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488386 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-model-cache\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488460    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-dshm\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tls-certs\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488598    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-home\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488783 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488644    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tmp-dir\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488783 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488677    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-model-cache\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.488783 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.488736    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kserve-provision-location\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.490790 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.490769    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-dshm\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.490915 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.490864    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tls-certs\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.499650 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.499627    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2ds7\" (UniqueName: \"kubernetes.io/projected/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kube-api-access-c2ds7\") pod \"prestop-hook-test-kserve-789646768b-swqvb\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.597131 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.597094    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:03.764563 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:03.764368    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"]
Jul 09 10:34:03.767171 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:34:03.767147    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f73b21_8baa_420e_9aa0_314c6d8b165d.slice/crio-fffcbbe7be28fda2a987425e97e6042f4640db0d4f7e0f29ee85bbccf2dbdfdd WatchSource:0}: Error finding container fffcbbe7be28fda2a987425e97e6042f4640db0d4f7e0f29ee85bbccf2dbdfdd: Status 404 returned error can't find the container with id fffcbbe7be28fda2a987425e97e6042f4640db0d4f7e0f29ee85bbccf2dbdfdd
Jul 09 10:34:04.227806 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:04.227770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb" event={"ID":"d6f73b21-8baa-420e-9aa0-314c6d8b165d","Type":"ContainerStarted","Data":"9a4a35f1095146f9268c49cc8cff1d3cd3ebd03749655b452d15cd6f07f30431"}
Jul 09 10:34:04.228193 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:04.227820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb" event={"ID":"d6f73b21-8baa-420e-9aa0-314c6d8b165d","Type":"ContainerStarted","Data":"fffcbbe7be28fda2a987425e97e6042f4640db0d4f7e0f29ee85bbccf2dbdfdd"}
Jul 09 10:34:08.245614 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:08.245582    2581 generic.go:356] "Generic (PLEG): container finished" podID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" containerID="9a4a35f1095146f9268c49cc8cff1d3cd3ebd03749655b452d15cd6f07f30431" exitCode=0
Jul 09 10:34:08.246012 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:08.245655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb" event={"ID":"d6f73b21-8baa-420e-9aa0-314c6d8b165d","Type":"ContainerDied","Data":"9a4a35f1095146f9268c49cc8cff1d3cd3ebd03749655b452d15cd6f07f30431"}
Jul 09 10:34:09.251509 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:09.251472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb" event={"ID":"d6f73b21-8baa-420e-9aa0-314c6d8b165d","Type":"ContainerStarted","Data":"b3acc9855d4ac29d7aed5da5c006250bfece679531b3d7d67a4f3e5f2ab72866"}
Jul 09 10:34:09.290229 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:09.290155    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb" podStartSLOduration=2.396231236 podStartE2EDuration="6.290133749s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.893902513s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:34:09.28996357 +0000 UTC m=+3641.614925855" watchObservedRunningTime="2026-07-09 10:34:09.290133749 +0000 UTC m=+3641.615096025"
Jul 09 10:34:09.597791 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:09.597704    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:13.597312 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:13.597274    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:34:13.598810 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:13.598779    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb" podUID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 09 10:34:18.223340 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:18.223294    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" containerID="cri-o://7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431" gracePeriod=45
Jul 09 10:34:19.002692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.002667    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:34:19.116212 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.116115    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-kserve-provision-location\") pod \"668b7870-23e5-4385-9d61-30a6461083c5\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") "
Jul 09 10:34:19.116212 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.116151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-dshm\") pod \"668b7870-23e5-4385-9d61-30a6461083c5\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") "
Jul 09 10:34:19.116212 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.116170    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-home\") pod \"668b7870-23e5-4385-9d61-30a6461083c5\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") "
Jul 09 10:34:19.116212 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.116188    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-model-cache\") pod \"668b7870-23e5-4385-9d61-30a6461083c5\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") "
Jul 09 10:34:19.116554 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.116217    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668b7870-23e5-4385-9d61-30a6461083c5-tls-certs\") pod \"668b7870-23e5-4385-9d61-30a6461083c5\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") "
Jul 09 10:34:19.116554 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.116240    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cwwr\" (UniqueName: \"kubernetes.io/projected/668b7870-23e5-4385-9d61-30a6461083c5-kube-api-access-6cwwr\") pod \"668b7870-23e5-4385-9d61-30a6461083c5\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") "
Jul 09 10:34:19.116554 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.116280    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-tmp-dir\") pod \"668b7870-23e5-4385-9d61-30a6461083c5\" (UID: \"668b7870-23e5-4385-9d61-30a6461083c5\") "
Jul 09 10:34:19.116554 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.116465    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-model-cache" (OuterVolumeSpecName: "model-cache") pod "668b7870-23e5-4385-9d61-30a6461083c5" (UID: "668b7870-23e5-4385-9d61-30a6461083c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:34:19.116554 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.116526    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:19.116925 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.116866    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-home" (OuterVolumeSpecName: "home") pod "668b7870-23e5-4385-9d61-30a6461083c5" (UID: "668b7870-23e5-4385-9d61-30a6461083c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:34:19.118753 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.118710    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-dshm" (OuterVolumeSpecName: "dshm") pod "668b7870-23e5-4385-9d61-30a6461083c5" (UID: "668b7870-23e5-4385-9d61-30a6461083c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:34:19.119767 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.119729    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/668b7870-23e5-4385-9d61-30a6461083c5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "668b7870-23e5-4385-9d61-30a6461083c5" (UID: "668b7870-23e5-4385-9d61-30a6461083c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:34:19.119869 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.119829    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/668b7870-23e5-4385-9d61-30a6461083c5-kube-api-access-6cwwr" (OuterVolumeSpecName: "kube-api-access-6cwwr") pod "668b7870-23e5-4385-9d61-30a6461083c5" (UID: "668b7870-23e5-4385-9d61-30a6461083c5"). InnerVolumeSpecName "kube-api-access-6cwwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:34:19.128801 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.128781    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "668b7870-23e5-4385-9d61-30a6461083c5" (UID: "668b7870-23e5-4385-9d61-30a6461083c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:34:19.180700 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.180652    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "668b7870-23e5-4385-9d61-30a6461083c5" (UID: "668b7870-23e5-4385-9d61-30a6461083c5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:34:19.217293 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.217260    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:19.217293 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.217287    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/668b7870-23e5-4385-9d61-30a6461083c5-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:19.217293 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.217299    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cwwr\" (UniqueName: \"kubernetes.io/projected/668b7870-23e5-4385-9d61-30a6461083c5-kube-api-access-6cwwr\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:19.217506 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.217332    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:19.217506 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.217343    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:19.217506 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.217350    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/668b7870-23e5-4385-9d61-30a6461083c5-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:19.295848 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.295813    2581 generic.go:356] "Generic (PLEG): container finished" podID="668b7870-23e5-4385-9d61-30a6461083c5" containerID="7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431" exitCode=0
Jul 09 10:34:19.296256 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.295860    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"
Jul 09 10:34:19.296256 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.295869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" event={"ID":"668b7870-23e5-4385-9d61-30a6461083c5","Type":"ContainerDied","Data":"7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431"}
Jul 09 10:34:19.296256 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.295950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" event={"ID":"668b7870-23e5-4385-9d61-30a6461083c5","Type":"ContainerDied","Data":"34b340b131d67d8fad69638d135c9db7969b8252527744e012437d3d6cc783b7"}
Jul 09 10:34:19.296256 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.295985    2581 scope.go:117] "RemoveContainer" containerID="7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431"
Jul 09 10:34:19.304972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.304952    2581 scope.go:117] "RemoveContainer" containerID="d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283"
Jul 09 10:34:19.315034 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.315019    2581 scope.go:117] "RemoveContainer" containerID="7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431"
Jul 09 10:34:19.315284 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:34:19.315266    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431\": container with ID starting with 7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431 not found: ID does not exist" containerID="7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431"
Jul 09 10:34:19.315376 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.315294    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431"} err="failed to get container status \"7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431\": rpc error: code = NotFound desc = could not find container \"7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431\": container with ID starting with 7df9896ffd4e3d924bef6b5f784a5c8531ff0d6cc843c46f434730fe077c4431 not found: ID does not exist"
Jul 09 10:34:19.315376 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.315316    2581 scope.go:117] "RemoveContainer" containerID="d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283"
Jul 09 10:34:19.315557 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:34:19.315541    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283\": container with ID starting with d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283 not found: ID does not exist" containerID="d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283"
Jul 09 10:34:19.315599 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.315563    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283"} err="failed to get container status \"d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283\": rpc error: code = NotFound desc = could not find container \"d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283\": container with ID starting with d7461ea3185804e36453226871d586271502299adca9e69ae52dfd8c5b650283 not found: ID does not exist"
Jul 09 10:34:19.323086 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.322882    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"]
Jul 09 10:34:19.327825 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.327804    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj"]
Jul 09 10:34:19.956783 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:19.956742    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-vjgdj" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 09 10:34:20.228473 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:20.228399    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="668b7870-23e5-4385-9d61-30a6461083c5" path="/var/lib/kubelet/pods/668b7870-23e5-4385-9d61-30a6461083c5/volumes"
Jul 09 10:34:23.597378 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:23.597341    2581 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb" podUID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 09 10:34:24.338845 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:24.338816    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"]
Jul 09 10:34:30.875343 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:30.875303    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 10:34:39.361363 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:39.361304    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb" podUID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" containerName="main" containerID="cri-o://b3acc9855d4ac29d7aed5da5c006250bfece679531b3d7d67a4f3e5f2ab72866" gracePeriod=45
Jul 09 10:34:45.897879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:45.897836    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" containerID="cri-o://31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666" gracePeriod=45
Jul 09 10:34:45.974265 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:45.974221    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 09 10:34:46.662150 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.662126    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:34:46.760588 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.760554    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-dshm\") pod \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") "
Jul 09 10:34:46.760768 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.760609    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tmp-dir\") pod \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") "
Jul 09 10:34:46.760768 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.760626    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kserve-provision-location\") pod \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") "
Jul 09 10:34:46.760768 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.760646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tls-certs\") pod \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") "
Jul 09 10:34:46.760768 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.760682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-home\") pod \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") "
Jul 09 10:34:46.760768 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.760723    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjhh9\" (UniqueName: \"kubernetes.io/projected/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kube-api-access-fjhh9\") pod \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") "
Jul 09 10:34:46.760768 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.760760    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-model-cache\") pod \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\" (UID: \"f6fca231-0c40-44e2-80fe-5cb7ce60b07e\") "
Jul 09 10:34:46.761219 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.761194    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f6fca231-0c40-44e2-80fe-5cb7ce60b07e" (UID: "f6fca231-0c40-44e2-80fe-5cb7ce60b07e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:34:46.761600 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.761571    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-home" (OuterVolumeSpecName: "home") pod "f6fca231-0c40-44e2-80fe-5cb7ce60b07e" (UID: "f6fca231-0c40-44e2-80fe-5cb7ce60b07e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:34:46.762881 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.762851    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-dshm" (OuterVolumeSpecName: "dshm") pod "f6fca231-0c40-44e2-80fe-5cb7ce60b07e" (UID: "f6fca231-0c40-44e2-80fe-5cb7ce60b07e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:34:46.763185 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.763163    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f6fca231-0c40-44e2-80fe-5cb7ce60b07e" (UID: "f6fca231-0c40-44e2-80fe-5cb7ce60b07e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:34:46.763481 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.763460    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kube-api-access-fjhh9" (OuterVolumeSpecName: "kube-api-access-fjhh9") pod "f6fca231-0c40-44e2-80fe-5cb7ce60b07e" (UID: "f6fca231-0c40-44e2-80fe-5cb7ce60b07e"). InnerVolumeSpecName "kube-api-access-fjhh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:34:46.772475 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.772450    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f6fca231-0c40-44e2-80fe-5cb7ce60b07e" (UID: "f6fca231-0c40-44e2-80fe-5cb7ce60b07e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:34:46.843266 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.843210    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6fca231-0c40-44e2-80fe-5cb7ce60b07e" (UID: "f6fca231-0c40-44e2-80fe-5cb7ce60b07e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:34:46.861638 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.861605    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:46.861638 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.861629    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:46.861638 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.861638    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:46.861847 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.861649    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:46.861847 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.861658    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:46.861847 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.861667    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:46.861847 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:46.861674    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjhh9\" (UniqueName: \"kubernetes.io/projected/f6fca231-0c40-44e2-80fe-5cb7ce60b07e-kube-api-access-fjhh9\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:34:47.416536 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.416473    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerID="31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666" exitCode=0
Jul 09 10:34:47.417000 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.416541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f6fca231-0c40-44e2-80fe-5cb7ce60b07e","Type":"ContainerDied","Data":"31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666"}
Jul 09 10:34:47.417000 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.416595    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"f6fca231-0c40-44e2-80fe-5cb7ce60b07e","Type":"ContainerDied","Data":"af7a26e89e50f4e15217be7682b19077276d3916208ea3ba9ef41c7b18d01ebd"}
Jul 09 10:34:47.417000 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.416545    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 09 10:34:47.417000 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.416627    2581 scope.go:117] "RemoveContainer" containerID="31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666"
Jul 09 10:34:47.425861 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.425843    2581 scope.go:117] "RemoveContainer" containerID="39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15"
Jul 09 10:34:47.433247 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.433228    2581 scope.go:117] "RemoveContainer" containerID="31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666"
Jul 09 10:34:47.433531 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:34:47.433511    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666\": container with ID starting with 31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666 not found: ID does not exist" containerID="31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666"
Jul 09 10:34:47.433617 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.433543    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666"} err="failed to get container status \"31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666\": rpc error: code = NotFound desc = could not find container \"31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666\": container with ID starting with 31c6c42b02c2eac07a4b9ac616c8c6fcfa023082c67ba6e352b22ac16c363666 not found: ID does not exist"
Jul 09 10:34:47.433617 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.433565    2581 scope.go:117] "RemoveContainer" containerID="39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15"
Jul 09 10:34:47.433822 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:34:47.433803    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15\": container with ID starting with 39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15 not found: ID does not exist" containerID="39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15"
Jul 09 10:34:47.433869 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.433831    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15"} err="failed to get container status \"39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15\": rpc error: code = NotFound desc = could not find container \"39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15\": container with ID starting with 39f8325d531d152a795fdc9ad860c755350073d30d2937ad7ee06562eff26c15 not found: ID does not exist"
Jul 09 10:34:47.449425 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.449402    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 10:34:47.455182 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.455163    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 09 10:34:47.478450 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:47.478424    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 10:34:48.228586 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:34:48.228549    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" path="/var/lib/kubelet/pods/f6fca231-0c40-44e2-80fe-5cb7ce60b07e/volumes"
Jul 09 10:35:02.501131 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:02.501016    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="9260a465-e723-4f54-8635-41793ab8f09c" containerName="main" containerID="cri-o://d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4" gracePeriod=45
Jul 09 10:35:03.255660 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.255633    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:35:03.298497 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.298406    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-model-cache\") pod \"9260a465-e723-4f54-8635-41793ab8f09c\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") "
Jul 09 10:35:03.298497 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.298457    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9260a465-e723-4f54-8635-41793ab8f09c-tls-certs\") pod \"9260a465-e723-4f54-8635-41793ab8f09c\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") "
Jul 09 10:35:03.298497 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.298484    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfdmt\" (UniqueName: \"kubernetes.io/projected/9260a465-e723-4f54-8635-41793ab8f09c-kube-api-access-qfdmt\") pod \"9260a465-e723-4f54-8635-41793ab8f09c\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") "
Jul 09 10:35:03.298791 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.298502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-kserve-provision-location\") pod \"9260a465-e723-4f54-8635-41793ab8f09c\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") "
Jul 09 10:35:03.298791 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.298527    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-home\") pod \"9260a465-e723-4f54-8635-41793ab8f09c\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") "
Jul 09 10:35:03.298791 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.298550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-dshm\") pod \"9260a465-e723-4f54-8635-41793ab8f09c\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") "
Jul 09 10:35:03.298791 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.298571    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-tmp-dir\") pod \"9260a465-e723-4f54-8635-41793ab8f09c\" (UID: \"9260a465-e723-4f54-8635-41793ab8f09c\") "
Jul 09 10:35:03.298791 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.298695    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9260a465-e723-4f54-8635-41793ab8f09c" (UID: "9260a465-e723-4f54-8635-41793ab8f09c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:35:03.298791 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.298771    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:03.299430 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.299397    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-home" (OuterVolumeSpecName: "home") pod "9260a465-e723-4f54-8635-41793ab8f09c" (UID: "9260a465-e723-4f54-8635-41793ab8f09c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:35:03.301232 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.301198    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9260a465-e723-4f54-8635-41793ab8f09c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9260a465-e723-4f54-8635-41793ab8f09c" (UID: "9260a465-e723-4f54-8635-41793ab8f09c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:35:03.301350 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.301260    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-dshm" (OuterVolumeSpecName: "dshm") pod "9260a465-e723-4f54-8635-41793ab8f09c" (UID: "9260a465-e723-4f54-8635-41793ab8f09c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:35:03.301569 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.301550    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9260a465-e723-4f54-8635-41793ab8f09c-kube-api-access-qfdmt" (OuterVolumeSpecName: "kube-api-access-qfdmt") pod "9260a465-e723-4f54-8635-41793ab8f09c" (UID: "9260a465-e723-4f54-8635-41793ab8f09c"). InnerVolumeSpecName "kube-api-access-qfdmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:35:03.320102 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.320070    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9260a465-e723-4f54-8635-41793ab8f09c" (UID: "9260a465-e723-4f54-8635-41793ab8f09c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:35:03.376547 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.376491    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9260a465-e723-4f54-8635-41793ab8f09c" (UID: "9260a465-e723-4f54-8635-41793ab8f09c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:35:03.399981 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.399953    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9260a465-e723-4f54-8635-41793ab8f09c-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:03.399981 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.399981    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdmt\" (UniqueName: \"kubernetes.io/projected/9260a465-e723-4f54-8635-41793ab8f09c-kube-api-access-qfdmt\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:03.400142 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.399990    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:03.400142 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.400000    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:03.400142 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.400017    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:03.400142 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.400024    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9260a465-e723-4f54-8635-41793ab8f09c-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:03.486138 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.486103    2581 generic.go:356] "Generic (PLEG): container finished" podID="9260a465-e723-4f54-8635-41793ab8f09c" containerID="d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4" exitCode=0
Jul 09 10:35:03.486138 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.486150    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 09 10:35:03.486365 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.486187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"9260a465-e723-4f54-8635-41793ab8f09c","Type":"ContainerDied","Data":"d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4"}
Jul 09 10:35:03.486365 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.486241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"9260a465-e723-4f54-8635-41793ab8f09c","Type":"ContainerDied","Data":"5991174c31f6e03bddea0df1c390ecee6aef256fdd237108fcdac406b21949d4"}
Jul 09 10:35:03.486365 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.486272    2581 scope.go:117] "RemoveContainer" containerID="d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4"
Jul 09 10:35:03.495145 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.495122    2581 scope.go:117] "RemoveContainer" containerID="0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421"
Jul 09 10:35:03.505666 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.505492    2581 scope.go:117] "RemoveContainer" containerID="d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4"
Jul 09 10:35:03.505920 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:35:03.505784    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4\": container with ID starting with d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4 not found: ID does not exist" containerID="d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4"
Jul 09 10:35:03.505920 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.505809    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4"} err="failed to get container status \"d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4\": rpc error: code = NotFound desc = could not find container \"d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4\": container with ID starting with d31f7bd175a12e0dc74f7cfb04236e350f69d55572103859497cb2eeb61ef4e4 not found: ID does not exist"
Jul 09 10:35:03.505920 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.505827    2581 scope.go:117] "RemoveContainer" containerID="0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421"
Jul 09 10:35:03.506129 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:35:03.506112    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421\": container with ID starting with 0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421 not found: ID does not exist" containerID="0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421"
Jul 09 10:35:03.506177 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.506135    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421"} err="failed to get container status \"0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421\": rpc error: code = NotFound desc = could not find container \"0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421\": container with ID starting with 0b96c70128dca238a7e7c7866b1ec730236c6e384e91967553a4b9f06cb19421 not found: ID does not exist"
Jul 09 10:35:03.509139 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.509118    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 10:35:03.511502 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:03.511478    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 09 10:35:04.227753 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:04.227719    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9260a465-e723-4f54-8635-41793ab8f09c" path="/var/lib/kubelet/pods/9260a465-e723-4f54-8635-41793ab8f09c/volumes"
Jul 09 10:35:24.574817 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:24.574787    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-789646768b-swqvb_d6f73b21-8baa-420e-9aa0-314c6d8b165d/main/0.log"
Jul 09 10:35:24.575350 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:24.575197    2581 generic.go:356] "Generic (PLEG): container finished" podID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" containerID="b3acc9855d4ac29d7aed5da5c006250bfece679531b3d7d67a4f3e5f2ab72866" exitCode=137
Jul 09 10:35:24.575350 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:24.575243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb" event={"ID":"d6f73b21-8baa-420e-9aa0-314c6d8b165d","Type":"ContainerDied","Data":"b3acc9855d4ac29d7aed5da5c006250bfece679531b3d7d67a4f3e5f2ab72866"}
Jul 09 10:35:24.993135 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:24.993113    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-789646768b-swqvb_d6f73b21-8baa-420e-9aa0-314c6d8b165d/main/0.log"
Jul 09 10:35:24.994735 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:24.994719    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:35:25.076961 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.076930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kserve-provision-location\") pod \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") "
Jul 09 10:35:25.077137 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.076982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-model-cache\") pod \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") "
Jul 09 10:35:25.077137 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.077013    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-dshm\") pod \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") "
Jul 09 10:35:25.077137 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.077034    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tmp-dir\") pod \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") "
Jul 09 10:35:25.077137 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.077056    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-home\") pod \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") "
Jul 09 10:35:25.077137 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.077088    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tls-certs\") pod \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") "
Jul 09 10:35:25.077405 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.077146    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2ds7\" (UniqueName: \"kubernetes.io/projected/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kube-api-access-c2ds7\") pod \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\" (UID: \"d6f73b21-8baa-420e-9aa0-314c6d8b165d\") "
Jul 09 10:35:25.077405 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.077296    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6f73b21-8baa-420e-9aa0-314c6d8b165d" (UID: "d6f73b21-8baa-420e-9aa0-314c6d8b165d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:35:25.077559 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.077473    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6f73b21-8baa-420e-9aa0-314c6d8b165d" (UID: "d6f73b21-8baa-420e-9aa0-314c6d8b165d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:35:25.077559 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.077498    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-home" (OuterVolumeSpecName: "home") pod "d6f73b21-8baa-420e-9aa0-314c6d8b165d" (UID: "d6f73b21-8baa-420e-9aa0-314c6d8b165d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:35:25.079222 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.079186    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6f73b21-8baa-420e-9aa0-314c6d8b165d" (UID: "d6f73b21-8baa-420e-9aa0-314c6d8b165d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:35:25.079327 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.079238    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-dshm" (OuterVolumeSpecName: "dshm") pod "d6f73b21-8baa-420e-9aa0-314c6d8b165d" (UID: "d6f73b21-8baa-420e-9aa0-314c6d8b165d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:35:25.079327 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.079272    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kube-api-access-c2ds7" (OuterVolumeSpecName: "kube-api-access-c2ds7") pod "d6f73b21-8baa-420e-9aa0-314c6d8b165d" (UID: "d6f73b21-8baa-420e-9aa0-314c6d8b165d"). InnerVolumeSpecName "kube-api-access-c2ds7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:35:25.135264 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.135193    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6f73b21-8baa-420e-9aa0-314c6d8b165d" (UID: "d6f73b21-8baa-420e-9aa0-314c6d8b165d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:35:25.177904 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.177870    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2ds7\" (UniqueName: \"kubernetes.io/projected/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kube-api-access-c2ds7\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:25.178024 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.177911    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-kserve-provision-location\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:25.178024 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.177922    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:25.178024 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.177931    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:25.178024 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.177959    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:25.178024 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.177967    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6f73b21-8baa-420e-9aa0-314c6d8b165d-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:25.178024 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.177975    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6f73b21-8baa-420e-9aa0-314c6d8b165d-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:35:25.580498 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.580463    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-789646768b-swqvb_d6f73b21-8baa-420e-9aa0-314c6d8b165d/main/0.log"
Jul 09 10:35:25.580972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.580856    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb" event={"ID":"d6f73b21-8baa-420e-9aa0-314c6d8b165d","Type":"ContainerDied","Data":"fffcbbe7be28fda2a987425e97e6042f4640db0d4f7e0f29ee85bbccf2dbdfdd"}
Jul 09 10:35:25.580972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.580866    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"
Jul 09 10:35:25.580972 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.580928    2581 scope.go:117] "RemoveContainer" containerID="b3acc9855d4ac29d7aed5da5c006250bfece679531b3d7d67a4f3e5f2ab72866"
Jul 09 10:35:25.589646 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.589627    2581 scope.go:117] "RemoveContainer" containerID="9a4a35f1095146f9268c49cc8cff1d3cd3ebd03749655b452d15cd6f07f30431"
Jul 09 10:35:25.617315 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.617293    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"]
Jul 09 10:35:25.622374 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:25.622352    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-789646768b-swqvb"]
Jul 09 10:35:26.227983 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:26.227956    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" path="/var/lib/kubelet/pods/d6f73b21-8baa-420e-9aa0-314c6d8b165d/volumes"
Jul 09 10:35:32.330329 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330298    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"]
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330562    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="storage-initializer"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330573    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="storage-initializer"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330590    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330598    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330609    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330614    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330622    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9260a465-e723-4f54-8635-41793ab8f09c" containerName="storage-initializer"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330627    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9260a465-e723-4f54-8635-41793ab8f09c" containerName="storage-initializer"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330634    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9260a465-e723-4f54-8635-41793ab8f09c" containerName="main"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330639    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9260a465-e723-4f54-8635-41793ab8f09c" containerName="main"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330645    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" containerName="storage-initializer"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330650    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" containerName="storage-initializer"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330655    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" containerName="main"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330662    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" containerName="main"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330671    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="storage-initializer"
Jul 09 10:35:32.330692 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330676    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="storage-initializer"
Jul 09 10:35:32.331203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330719    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="668b7870-23e5-4385-9d61-30a6461083c5" containerName="main"
Jul 09 10:35:32.331203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330728    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f73b21-8baa-420e-9aa0-314c6d8b165d" containerName="main"
Jul 09 10:35:32.331203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330733    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6fca231-0c40-44e2-80fe-5cb7ce60b07e" containerName="main"
Jul 09 10:35:32.331203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.330741    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9260a465-e723-4f54-8635-41793ab8f09c" containerName="main"
Jul 09 10:35:32.333870 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.333853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.338373 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.338350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkhz8\""
Jul 09 10:35:32.338510 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.338351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 10:35:32.338510 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.338389    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 09 10:35:32.338510 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.338404    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 10:35:32.346961 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.346939    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"]
Jul 09 10:35:32.436265 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.436234    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.436265 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.436268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.436508 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.436292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.436508 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.436312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.436508 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.436426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382d5c3e-c7fe-4827-892d-bc21dac47c01-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.436508 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.436474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rqw\" (UniqueName: \"kubernetes.io/projected/382d5c3e-c7fe-4827-892d-bc21dac47c01-kube-api-access-l4rqw\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.537430 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.537397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382d5c3e-c7fe-4827-892d-bc21dac47c01-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.537605 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.537434    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rqw\" (UniqueName: \"kubernetes.io/projected/382d5c3e-c7fe-4827-892d-bc21dac47c01-kube-api-access-l4rqw\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.537605 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.537458    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.537605 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.537483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.537786 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.537606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.537786 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.537649    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.537926 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.537831    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-home\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.537926 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.537862    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-tmp-dir\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.538021 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.537985    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-model-cache\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.539800 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.539779    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-dshm\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.540035 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.540016    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382d5c3e-c7fe-4827-892d-bc21dac47c01-tls-certs\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.549182 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.549163    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rqw\" (UniqueName: \"kubernetes.io/projected/382d5c3e-c7fe-4827-892d-bc21dac47c01-kube-api-access-l4rqw\") pod \"rolling-upgrade-test-kserve-54bc76f55b-7whj9\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.614997 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.614927    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"]
Jul 09 10:35:32.617140 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.617124    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:32.623542 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.623521    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-epp-sa-dockercfg-zxqb2\""
Jul 09 10:35:32.643298 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.643277    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:32.645666 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.645649    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"]
Jul 09 10:35:32.739710 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.739679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:32.739847 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.739733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:32.739847 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.739797    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tq8n\" (UniqueName: \"kubernetes.io/projected/1caf6ed8-4c51-4a62-9883-6dd38c74f587-kube-api-access-5tq8n\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:32.774615 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.774582    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"]
Jul 09 10:35:32.777374 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:35:32.777348    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod382d5c3e_c7fe_4827_892d_bc21dac47c01.slice/crio-9e27d8da4a603737e41b6a519686047058bdf966c8eb3012736ff5827ec90b52 WatchSource:0}: Error finding container 9e27d8da4a603737e41b6a519686047058bdf966c8eb3012736ff5827ec90b52: Status 404 returned error can't find the container with id 9e27d8da4a603737e41b6a519686047058bdf966c8eb3012736ff5827ec90b52
Jul 09 10:35:32.840559 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.840538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:32.840660 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.840571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tq8n\" (UniqueName: \"kubernetes.io/projected/1caf6ed8-4c51-4a62-9883-6dd38c74f587-kube-api-access-5tq8n\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:32.840660 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.840632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:32.840968 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.840947    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:32.842874 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.842854    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:32.868827 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.868766    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tq8n\" (UniqueName: \"kubernetes.io/projected/1caf6ed8-4c51-4a62-9883-6dd38c74f587-kube-api-access-5tq8n\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:32.925925 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:32.925874    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:33.068308 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:33.068277    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"]
Jul 09 10:35:33.069154 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:35:33.069131    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caf6ed8_4c51_4a62_9883_6dd38c74f587.slice/crio-27d7086d54cc3670164419f8cd9d42a0ee8adaa3ad6a88d80293434b729b17a5 WatchSource:0}: Error finding container 27d7086d54cc3670164419f8cd9d42a0ee8adaa3ad6a88d80293434b729b17a5: Status 404 returned error can't find the container with id 27d7086d54cc3670164419f8cd9d42a0ee8adaa3ad6a88d80293434b729b17a5
Jul 09 10:35:33.613585 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:33.613548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm" event={"ID":"1caf6ed8-4c51-4a62-9883-6dd38c74f587","Type":"ContainerStarted","Data":"8a18b70688edb9c3e1d1172a583ea4ffbc4c221d306a423b636477ea4402ddde"}
Jul 09 10:35:33.614037 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:33.613614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm" event={"ID":"1caf6ed8-4c51-4a62-9883-6dd38c74f587","Type":"ContainerStarted","Data":"27d7086d54cc3670164419f8cd9d42a0ee8adaa3ad6a88d80293434b729b17a5"}
Jul 09 10:35:33.614037 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:33.613642    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:35:33.615091 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:33.615071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9" event={"ID":"382d5c3e-c7fe-4827-892d-bc21dac47c01","Type":"ContainerStarted","Data":"7ba1982dc698d5352e82f7393b46ec04021bbcb01f5890a771f1ef5b9a7cb52d"}
Jul 09 10:35:33.615184 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:33.615107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9" event={"ID":"382d5c3e-c7fe-4827-892d-bc21dac47c01","Type":"ContainerStarted","Data":"9e27d8da4a603737e41b6a519686047058bdf966c8eb3012736ff5827ec90b52"}
Jul 09 10:35:33.644013 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:33.643981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:33.687504 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:33.687461    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm" podStartSLOduration=1.687446475 podStartE2EDuration="1.687446475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:35:33.650508533 +0000 UTC m=+3725.975470818" watchObservedRunningTime="2026-07-09 10:35:33.687446475 +0000 UTC m=+3726.012408758"
Jul 09 10:35:33.687811 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:33.687793    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9" podStartSLOduration=1.687788193 podStartE2EDuration="1.687788193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:35:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:35:33.685697463 +0000 UTC m=+3726.010659748" watchObservedRunningTime="2026-07-09 10:35:33.687788193 +0000 UTC m=+3726.012750476"
Jul 09 10:35:42.644264 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:42.644221    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:42.657105 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:42.657077    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:35:42.668201 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:35:42.668174    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:36:04.620577 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:04.620548    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:36:16.602911 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:16.602856    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"]
Jul 09 10:36:22.639237 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.639151    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"]
Jul 09 10:36:22.648392 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:36:22.648360    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 10:36:22.648392 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 10:36:22.648392 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 10:36:22.648392 ip-10-0-129-157 kubenswrapper[2581]:  > containerID="8a18b70688edb9c3e1d1172a583ea4ffbc4c221d306a423b636477ea4402ddde" cmd=["/bin/sleep","15"]
Jul 09 10:36:22.648686 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.648412    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm" message="" err=<
Jul 09 10:36:22.648686 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 10:36:22.648686 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 10:36:22.648686 ip-10-0-129-157 kubenswrapper[2581]:  >
Jul 09 10:36:22.648686 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:36:22.648428    2581 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 09 10:36:22.648686 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 10:36:22.648686 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 10:36:22.648686 ip-10-0-129-157 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm" podUID="1caf6ed8-4c51-4a62-9883-6dd38c74f587" containerName="main" containerID="cri-o://8a18b70688edb9c3e1d1172a583ea4ffbc4c221d306a423b636477ea4402ddde"
Jul 09 10:36:22.648686 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.648468    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm" podUID="1caf6ed8-4c51-4a62-9883-6dd38c74f587" containerName="main" containerID="cri-o://8a18b70688edb9c3e1d1172a583ea4ffbc4c221d306a423b636477ea4402ddde" gracePeriod=60
Jul 09 10:36:22.802347 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.802311    2581 generic.go:356] "Generic (PLEG): container finished" podID="1caf6ed8-4c51-4a62-9883-6dd38c74f587" containerID="8a18b70688edb9c3e1d1172a583ea4ffbc4c221d306a423b636477ea4402ddde" exitCode=0
Jul 09 10:36:22.802514 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.802382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm" event={"ID":"1caf6ed8-4c51-4a62-9883-6dd38c74f587","Type":"ContainerDied","Data":"8a18b70688edb9c3e1d1172a583ea4ffbc4c221d306a423b636477ea4402ddde"}
Jul 09 10:36:22.893074 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.893021    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:36:22.956040 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.956015    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tq8n\" (UniqueName: \"kubernetes.io/projected/1caf6ed8-4c51-4a62-9883-6dd38c74f587-kube-api-access-5tq8n\") pod \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") "
Jul 09 10:36:22.956171 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.956058    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tokenizer-uds\") pod \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") "
Jul 09 10:36:22.956171 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.956093    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tls-certs\") pod \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\" (UID: \"1caf6ed8-4c51-4a62-9883-6dd38c74f587\") "
Jul 09 10:36:22.956395 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.956368    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1caf6ed8-4c51-4a62-9883-6dd38c74f587" (UID: "1caf6ed8-4c51-4a62-9883-6dd38c74f587"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:36:22.958113 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.958094    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1caf6ed8-4c51-4a62-9883-6dd38c74f587" (UID: "1caf6ed8-4c51-4a62-9883-6dd38c74f587"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:36:22.958220 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:22.958205    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1caf6ed8-4c51-4a62-9883-6dd38c74f587-kube-api-access-5tq8n" (OuterVolumeSpecName: "kube-api-access-5tq8n") pod "1caf6ed8-4c51-4a62-9883-6dd38c74f587" (UID: "1caf6ed8-4c51-4a62-9883-6dd38c74f587"). InnerVolumeSpecName "kube-api-access-5tq8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:36:23.057016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:23.056990    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tokenizer-uds\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:36:23.057016 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:23.057013    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1caf6ed8-4c51-4a62-9883-6dd38c74f587-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:36:23.057171 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:23.057022    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tq8n\" (UniqueName: \"kubernetes.io/projected/1caf6ed8-4c51-4a62-9883-6dd38c74f587-kube-api-access-5tq8n\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:36:23.812260 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:23.812221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm" event={"ID":"1caf6ed8-4c51-4a62-9883-6dd38c74f587","Type":"ContainerDied","Data":"27d7086d54cc3670164419f8cd9d42a0ee8adaa3ad6a88d80293434b729b17a5"}
Jul 09 10:36:23.812260 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:23.812262    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"
Jul 09 10:36:23.812791 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:23.812307    2581 scope.go:117] "RemoveContainer" containerID="8a18b70688edb9c3e1d1172a583ea4ffbc4c221d306a423b636477ea4402ddde"
Jul 09 10:36:23.847329 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:23.847307    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"]
Jul 09 10:36:23.855078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:23.855059    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-kkznm"]
Jul 09 10:36:24.227538 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:24.227509    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1caf6ed8-4c51-4a62-9883-6dd38c74f587" path="/var/lib/kubelet/pods/1caf6ed8-4c51-4a62-9883-6dd38c74f587/volumes"
Jul 09 10:36:31.623023 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.622984    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9" podUID="382d5c3e-c7fe-4827-892d-bc21dac47c01" containerName="main" containerID="cri-o://7ba1982dc698d5352e82f7393b46ec04021bbcb01f5890a771f1ef5b9a7cb52d" gracePeriod=45
Jul 09 10:36:31.658001 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.657974    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9" podUID="382d5c3e-c7fe-4827-892d-bc21dac47c01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 09 10:36:31.842854 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.842824    2581 generic.go:356] "Generic (PLEG): container finished" podID="382d5c3e-c7fe-4827-892d-bc21dac47c01" containerID="7ba1982dc698d5352e82f7393b46ec04021bbcb01f5890a771f1ef5b9a7cb52d" exitCode=0
Jul 09 10:36:31.843043 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.842864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9" event={"ID":"382d5c3e-c7fe-4827-892d-bc21dac47c01","Type":"ContainerDied","Data":"7ba1982dc698d5352e82f7393b46ec04021bbcb01f5890a771f1ef5b9a7cb52d"}
Jul 09 10:36:31.866051 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.866030    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:36:31.925439 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.925363    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4rqw\" (UniqueName: \"kubernetes.io/projected/382d5c3e-c7fe-4827-892d-bc21dac47c01-kube-api-access-l4rqw\") pod \"382d5c3e-c7fe-4827-892d-bc21dac47c01\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") "
Jul 09 10:36:31.925439 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.925409    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-model-cache\") pod \"382d5c3e-c7fe-4827-892d-bc21dac47c01\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") "
Jul 09 10:36:31.925439 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.925430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-home\") pod \"382d5c3e-c7fe-4827-892d-bc21dac47c01\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") "
Jul 09 10:36:31.925678 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.925503    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382d5c3e-c7fe-4827-892d-bc21dac47c01-tls-certs\") pod \"382d5c3e-c7fe-4827-892d-bc21dac47c01\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") "
Jul 09 10:36:31.925678 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.925521    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-tmp-dir\") pod \"382d5c3e-c7fe-4827-892d-bc21dac47c01\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") "
Jul 09 10:36:31.925678 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.925557    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-dshm\") pod \"382d5c3e-c7fe-4827-892d-bc21dac47c01\" (UID: \"382d5c3e-c7fe-4827-892d-bc21dac47c01\") "
Jul 09 10:36:31.925834 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.925701    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-model-cache" (OuterVolumeSpecName: "model-cache") pod "382d5c3e-c7fe-4827-892d-bc21dac47c01" (UID: "382d5c3e-c7fe-4827-892d-bc21dac47c01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:36:31.925834 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.925725    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-home" (OuterVolumeSpecName: "home") pod "382d5c3e-c7fe-4827-892d-bc21dac47c01" (UID: "382d5c3e-c7fe-4827-892d-bc21dac47c01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:36:31.925957 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.925842    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "382d5c3e-c7fe-4827-892d-bc21dac47c01" (UID: "382d5c3e-c7fe-4827-892d-bc21dac47c01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:36:31.927558 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.927528    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-dshm" (OuterVolumeSpecName: "dshm") pod "382d5c3e-c7fe-4827-892d-bc21dac47c01" (UID: "382d5c3e-c7fe-4827-892d-bc21dac47c01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:36:31.927662 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.927538    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382d5c3e-c7fe-4827-892d-bc21dac47c01-kube-api-access-l4rqw" (OuterVolumeSpecName: "kube-api-access-l4rqw") pod "382d5c3e-c7fe-4827-892d-bc21dac47c01" (UID: "382d5c3e-c7fe-4827-892d-bc21dac47c01"). InnerVolumeSpecName "kube-api-access-l4rqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:36:31.927662 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:31.927563    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/382d5c3e-c7fe-4827-892d-bc21dac47c01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "382d5c3e-c7fe-4827-892d-bc21dac47c01" (UID: "382d5c3e-c7fe-4827-892d-bc21dac47c01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:36:32.026273 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.026242    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:36:32.026273 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.026270    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:36:32.026273 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.026278    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/382d5c3e-c7fe-4827-892d-bc21dac47c01-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:36:32.026505 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.026285    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:36:32.026505 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.026294    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/382d5c3e-c7fe-4827-892d-bc21dac47c01-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:36:32.026505 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.026303    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4rqw\" (UniqueName: \"kubernetes.io/projected/382d5c3e-c7fe-4827-892d-bc21dac47c01-kube-api-access-l4rqw\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:36:32.847104 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.847076    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"
Jul 09 10:36:32.847104 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.847100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9" event={"ID":"382d5c3e-c7fe-4827-892d-bc21dac47c01","Type":"ContainerDied","Data":"9e27d8da4a603737e41b6a519686047058bdf966c8eb3012736ff5827ec90b52"}
Jul 09 10:36:32.847541 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.847164    2581 scope.go:117] "RemoveContainer" containerID="7ba1982dc698d5352e82f7393b46ec04021bbcb01f5890a771f1ef5b9a7cb52d"
Jul 09 10:36:32.863667 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.863644    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"]
Jul 09 10:36:32.868634 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:32.868613    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-54bc76f55b-7whj9"]
Jul 09 10:36:34.227736 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:34.227697    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="382d5c3e-c7fe-4827-892d-bc21dac47c01" path="/var/lib/kubelet/pods/382d5c3e-c7fe-4827-892d-bc21dac47c01/volumes"
Jul 09 10:36:41.633790 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.633753    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-df5969cfc-jk8w2"]
Jul 09 10:36:41.634183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.634042    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1caf6ed8-4c51-4a62-9883-6dd38c74f587" containerName="main"
Jul 09 10:36:41.634183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.634053    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1caf6ed8-4c51-4a62-9883-6dd38c74f587" containerName="main"
Jul 09 10:36:41.634183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.634073    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="382d5c3e-c7fe-4827-892d-bc21dac47c01" containerName="main"
Jul 09 10:36:41.634183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.634078    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="382d5c3e-c7fe-4827-892d-bc21dac47c01" containerName="main"
Jul 09 10:36:41.634183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.634119    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="382d5c3e-c7fe-4827-892d-bc21dac47c01" containerName="main"
Jul 09 10:36:41.634183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.634128    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1caf6ed8-4c51-4a62-9883-6dd38c74f587" containerName="main"
Jul 09 10:36:41.636851 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.636833    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2"
Jul 09 10:36:41.640675 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.640654    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 09 10:36:41.640798 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.640676    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-wd9h7\""
Jul 09 10:36:41.651870 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.651849    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-df5969cfc-jk8w2"]
Jul 09 10:36:41.801382 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.801350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfvzb\" (UniqueName: \"kubernetes.io/projected/8187848a-035e-4601-9d07-7c242c78d4bc-kube-api-access-xfvzb\") pod \"llmisvc-controller-manager-df5969cfc-jk8w2\" (UID: \"8187848a-035e-4601-9d07-7c242c78d4bc\") " pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2"
Jul 09 10:36:41.801520 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.801392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8187848a-035e-4601-9d07-7c242c78d4bc-cert\") pod \"llmisvc-controller-manager-df5969cfc-jk8w2\" (UID: \"8187848a-035e-4601-9d07-7c242c78d4bc\") " pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2"
Jul 09 10:36:41.902051 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.901958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfvzb\" (UniqueName: \"kubernetes.io/projected/8187848a-035e-4601-9d07-7c242c78d4bc-kube-api-access-xfvzb\") pod \"llmisvc-controller-manager-df5969cfc-jk8w2\" (UID: \"8187848a-035e-4601-9d07-7c242c78d4bc\") " pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2"
Jul 09 10:36:41.902051 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.902024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8187848a-035e-4601-9d07-7c242c78d4bc-cert\") pod \"llmisvc-controller-manager-df5969cfc-jk8w2\" (UID: \"8187848a-035e-4601-9d07-7c242c78d4bc\") " pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2"
Jul 09 10:36:41.904443 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.904424    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8187848a-035e-4601-9d07-7c242c78d4bc-cert\") pod \"llmisvc-controller-manager-df5969cfc-jk8w2\" (UID: \"8187848a-035e-4601-9d07-7c242c78d4bc\") " pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2"
Jul 09 10:36:41.931128 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.931098    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfvzb\" (UniqueName: \"kubernetes.io/projected/8187848a-035e-4601-9d07-7c242c78d4bc-kube-api-access-xfvzb\") pod \"llmisvc-controller-manager-df5969cfc-jk8w2\" (UID: \"8187848a-035e-4601-9d07-7c242c78d4bc\") " pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2"
Jul 09 10:36:41.946075 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:41.946050    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2"
Jul 09 10:36:42.076712 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:42.076681    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-df5969cfc-jk8w2"]
Jul 09 10:36:42.077424 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:36:42.077391    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8187848a_035e_4601_9d07_7c242c78d4bc.slice/crio-b1c54979fe6f1a3a1ba8b3ef9ec9515f539543cd7d631230932377cd5183922e WatchSource:0}: Error finding container b1c54979fe6f1a3a1ba8b3ef9ec9515f539543cd7d631230932377cd5183922e: Status 404 returned error can't find the container with id b1c54979fe6f1a3a1ba8b3ef9ec9515f539543cd7d631230932377cd5183922e
Jul 09 10:36:42.078694 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:42.078677    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 10:36:42.885758 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:42.885728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2" event={"ID":"8187848a-035e-4601-9d07-7c242c78d4bc","Type":"ContainerStarted","Data":"b1c54979fe6f1a3a1ba8b3ef9ec9515f539543cd7d631230932377cd5183922e"}
Jul 09 10:36:45.897288 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:45.897257    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2" event={"ID":"8187848a-035e-4601-9d07-7c242c78d4bc","Type":"ContainerStarted","Data":"98e4fb6e171a3aea311d0fc583b319fae7d0af0eecc65842de8c6f30c96b0fbe"}
Jul 09 10:36:45.897676 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:45.897349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2"
Jul 09 10:36:45.916594 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:36:45.916544    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2" podStartSLOduration=1.927604136 podStartE2EDuration="4.916523104s" totalImagesPullingTime="2.988918968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:36:45.91511495 +0000 UTC m=+3798.240077245" watchObservedRunningTime="2026-07-09 10:36:45.916523104 +0000 UTC m=+3798.241485388"
Jul 09 10:37:16.903828 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:16.903799    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-df5969cfc-jk8w2"
Jul 09 10:37:47.766996 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.766964    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"]
Jul 09 10:37:47.770008 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.769992    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:47.773386 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.773361    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkhz8\""
Jul 09 10:37:47.773386 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.773379    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 10:37:47.773580 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.773379    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 09 10:37:47.773580 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.773486    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 10:37:47.784229 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.784204    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"]
Jul 09 10:37:47.908832 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.908798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:47.908832 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.908830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:47.909059 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.908867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e551f8-0c4b-4040-a803-892aef9a5bee-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:47.909059 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.908959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5z4\" (UniqueName: \"kubernetes.io/projected/b6e551f8-0c4b-4040-a803-892aef9a5bee-kube-api-access-gk5z4\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:47.909059 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.908993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:47.909059 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:47.909007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.009857 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.009812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.009857 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.009851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.010238 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.009925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e551f8-0c4b-4040-a803-892aef9a5bee-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.010238 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.009982    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5z4\" (UniqueName: \"kubernetes.io/projected/b6e551f8-0c4b-4040-a803-892aef9a5bee-kube-api-access-gk5z4\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.010238 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.010023    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.010238 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.010045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.010463 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.010387    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.010561 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.010538    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-home\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.010817 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.010794    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-model-cache\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.012655 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.012629    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e551f8-0c4b-4040-a803-892aef9a5bee-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.012756 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.012722    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-dshm\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.023900 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.023830    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5z4\" (UniqueName: \"kubernetes.io/projected/b6e551f8-0c4b-4040-a803-892aef9a5bee-kube-api-access-gk5z4\") pod \"scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") " pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.079689 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.079658    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:48.219332 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:48.219301    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"]
Jul 09 10:37:48.222722 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:37:48.222694    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e551f8_0c4b_4040_a803_892aef9a5bee.slice/crio-c3f13ab0256fc0ef4ad397e0a93b421de6be6a4e563a71938a1e1b16b7214382 WatchSource:0}: Error finding container c3f13ab0256fc0ef4ad397e0a93b421de6be6a4e563a71938a1e1b16b7214382: Status 404 returned error can't find the container with id c3f13ab0256fc0ef4ad397e0a93b421de6be6a4e563a71938a1e1b16b7214382
Jul 09 10:37:49.135433 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:49.134647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6" event={"ID":"b6e551f8-0c4b-4040-a803-892aef9a5bee","Type":"ContainerStarted","Data":"ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a"}
Jul 09 10:37:49.135433 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:49.134715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6" event={"ID":"b6e551f8-0c4b-4040-a803-892aef9a5bee","Type":"ContainerStarted","Data":"c3f13ab0256fc0ef4ad397e0a93b421de6be6a4e563a71938a1e1b16b7214382"}
Jul 09 10:37:49.157203 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:49.157152    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6" podStartSLOduration=2.157138962 podStartE2EDuration="2.157138962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:37:49.156090335 +0000 UTC m=+3861.481052619" watchObservedRunningTime="2026-07-09 10:37:49.157138962 +0000 UTC m=+3861.482101245"
Jul 09 10:37:50.080836 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:50.080761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:58.080446 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:58.080406    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:58.092443 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:58.092408    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:37:58.177640 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:37:58.177612    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:38:30.250439 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:30.250410    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:38:30.257807 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:30.257788    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:38:33.015678 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:33.015644    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"]
Jul 09 10:38:48.036265 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.036206    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6" podUID="b6e551f8-0c4b-4040-a803-892aef9a5bee" containerName="main" containerID="cri-o://ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a" gracePeriod=45
Jul 09 10:38:48.167498 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.167467    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6" podUID="b6e551f8-0c4b-4040-a803-892aef9a5bee" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 09 10:38:48.275429 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.275409    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:38:48.346494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.346425    2581 generic.go:356] "Generic (PLEG): container finished" podID="b6e551f8-0c4b-4040-a803-892aef9a5bee" containerID="ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a" exitCode=0
Jul 09 10:38:48.346494 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.346472    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"
Jul 09 10:38:48.346661 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.346498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6" event={"ID":"b6e551f8-0c4b-4040-a803-892aef9a5bee","Type":"ContainerDied","Data":"ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a"}
Jul 09 10:38:48.346661 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.346542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6" event={"ID":"b6e551f8-0c4b-4040-a803-892aef9a5bee","Type":"ContainerDied","Data":"c3f13ab0256fc0ef4ad397e0a93b421de6be6a4e563a71938a1e1b16b7214382"}
Jul 09 10:38:48.346661 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.346566    2581 scope.go:117] "RemoveContainer" containerID="ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a"
Jul 09 10:38:48.354218 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.354199    2581 scope.go:117] "RemoveContainer" containerID="ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a"
Jul 09 10:38:48.354482 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:38:48.354460    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a\": container with ID starting with ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a not found: ID does not exist" containerID="ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a"
Jul 09 10:38:48.354526 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.354492    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a"} err="failed to get container status \"ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a\": rpc error: code = NotFound desc = could not find container \"ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a\": container with ID starting with ed5b5a4f65d86a252892ce7cfe33e55c9b60517a6f0fb244e240414d47b6b04a not found: ID does not exist"
Jul 09 10:38:48.366965 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.366951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk5z4\" (UniqueName: \"kubernetes.io/projected/b6e551f8-0c4b-4040-a803-892aef9a5bee-kube-api-access-gk5z4\") pod \"b6e551f8-0c4b-4040-a803-892aef9a5bee\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") "
Jul 09 10:38:48.367030 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.366982    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-home\") pod \"b6e551f8-0c4b-4040-a803-892aef9a5bee\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") "
Jul 09 10:38:48.367030 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.367006    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-dshm\") pod \"b6e551f8-0c4b-4040-a803-892aef9a5bee\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") "
Jul 09 10:38:48.367030 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.367026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-tmp-dir\") pod \"b6e551f8-0c4b-4040-a803-892aef9a5bee\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") "
Jul 09 10:38:48.367147 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.367060    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-model-cache\") pod \"b6e551f8-0c4b-4040-a803-892aef9a5bee\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") "
Jul 09 10:38:48.367147 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.367098    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e551f8-0c4b-4040-a803-892aef9a5bee-tls-certs\") pod \"b6e551f8-0c4b-4040-a803-892aef9a5bee\" (UID: \"b6e551f8-0c4b-4040-a803-892aef9a5bee\") "
Jul 09 10:38:48.367274 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.367251    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-home" (OuterVolumeSpecName: "home") pod "b6e551f8-0c4b-4040-a803-892aef9a5bee" (UID: "b6e551f8-0c4b-4040-a803-892aef9a5bee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:38:48.367361 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.367267    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b6e551f8-0c4b-4040-a803-892aef9a5bee" (UID: "b6e551f8-0c4b-4040-a803-892aef9a5bee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:38:48.367361 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.367343    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:38:48.367474 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.367365    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:38:48.367474 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.367361    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-model-cache" (OuterVolumeSpecName: "model-cache") pod "b6e551f8-0c4b-4040-a803-892aef9a5bee" (UID: "b6e551f8-0c4b-4040-a803-892aef9a5bee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:38:48.369122 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.369099    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e551f8-0c4b-4040-a803-892aef9a5bee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b6e551f8-0c4b-4040-a803-892aef9a5bee" (UID: "b6e551f8-0c4b-4040-a803-892aef9a5bee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:38:48.369122 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.369111    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e551f8-0c4b-4040-a803-892aef9a5bee-kube-api-access-gk5z4" (OuterVolumeSpecName: "kube-api-access-gk5z4") pod "b6e551f8-0c4b-4040-a803-892aef9a5bee" (UID: "b6e551f8-0c4b-4040-a803-892aef9a5bee"). InnerVolumeSpecName "kube-api-access-gk5z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:38:48.369252 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.369108    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-dshm" (OuterVolumeSpecName: "dshm") pod "b6e551f8-0c4b-4040-a803-892aef9a5bee" (UID: "b6e551f8-0c4b-4040-a803-892aef9a5bee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:38:48.468333 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.468298    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6e551f8-0c4b-4040-a803-892aef9a5bee-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:38:48.468333 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.468328    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk5z4\" (UniqueName: \"kubernetes.io/projected/b6e551f8-0c4b-4040-a803-892aef9a5bee-kube-api-access-gk5z4\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:38:48.468333 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.468339    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:38:48.468568 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.468349    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6e551f8-0c4b-4040-a803-892aef9a5bee-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:38:48.671345 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.671316    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"]
Jul 09 10:38:48.682339 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:48.682312    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-configmap-ref-test-kserve-744bf59fbd-7flw6"]
Jul 09 10:38:50.227721 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:38:50.227679    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6e551f8-0c4b-4040-a803-892aef9a5bee" path="/var/lib/kubelet/pods/b6e551f8-0c4b-4040-a803-892aef9a5bee/volumes"
Jul 09 10:39:04.612899 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.612861    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"]
Jul 09 10:39:04.613404 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.613139    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6e551f8-0c4b-4040-a803-892aef9a5bee" containerName="main"
Jul 09 10:39:04.613404 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.613150    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6e551f8-0c4b-4040-a803-892aef9a5bee" containerName="main"
Jul 09 10:39:04.613404 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.613215    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6e551f8-0c4b-4040-a803-892aef9a5bee" containerName="main"
Jul 09 10:39:04.617154 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.617135    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.621550 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.621531    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkhz8\""
Jul 09 10:39:04.621674 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.621589    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 10:39:04.622537 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.622519    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 10:39:04.622640 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.622523    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 09 10:39:04.634936 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.634913    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"]
Jul 09 10:39:04.696770 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.696741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.696989 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.696793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9vhv\" (UniqueName: \"kubernetes.io/projected/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-kube-api-access-n9vhv\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.696989 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.696841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.696989 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.696935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.696989 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.696973    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.697133 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.697001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.797483 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.797451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.797630 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.797491    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.797630 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.797518    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.797630 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.797549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.797630 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.797577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.797630 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.797600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9vhv\" (UniqueName: \"kubernetes.io/projected/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-kube-api-access-n9vhv\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.797956 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.797931    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-model-cache\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.798055 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.798009    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.798055 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.798026    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-home\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.799775 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.799757    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-dshm\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.800069 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.800049    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.814501 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.814480    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9vhv\" (UniqueName: \"kubernetes.io/projected/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-kube-api-access-n9vhv\") pod \"scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:04.926949 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:04.926847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:05.077284 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.077250    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"]
Jul 09 10:39:05.080522 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.080505    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.084704 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.084685    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-gdjrd\""
Jul 09 10:39:05.097837 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.097812    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"]
Jul 09 10:39:05.103356 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:39:05.103334    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314c6992_1b3b_480a_8ccb_d2fe0e092ddd.slice/crio-7c387aec854ebdecf1f9e86cba6d75c9aae2382afcec0a231f4f7635b50e1533 WatchSource:0}: Error finding container 7c387aec854ebdecf1f9e86cba6d75c9aae2382afcec0a231f4f7635b50e1533: Status 404 returned error can't find the container with id 7c387aec854ebdecf1f9e86cba6d75c9aae2382afcec0a231f4f7635b50e1533
Jul 09 10:39:05.122540 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.122520    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"]
Jul 09 10:39:05.201538 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.201466    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhk2w\" (UniqueName: \"kubernetes.io/projected/d766ea3f-032c-425e-9438-15d4e7bbfa05-kube-api-access-bhk2w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.201538 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.201503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d766ea3f-032c-425e-9438-15d4e7bbfa05-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.201701 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.201566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d766ea3f-032c-425e-9438-15d4e7bbfa05-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.302684 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.302632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d766ea3f-032c-425e-9438-15d4e7bbfa05-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.302873 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.302712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhk2w\" (UniqueName: \"kubernetes.io/projected/d766ea3f-032c-425e-9438-15d4e7bbfa05-kube-api-access-bhk2w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.302873 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.302740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d766ea3f-032c-425e-9438-15d4e7bbfa05-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.303130 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.303112    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d766ea3f-032c-425e-9438-15d4e7bbfa05-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.305170 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.305144    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d766ea3f-032c-425e-9438-15d4e7bbfa05-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.323352 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.323328    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhk2w\" (UniqueName: \"kubernetes.io/projected/d766ea3f-032c-425e-9438-15d4e7bbfa05-kube-api-access-bhk2w\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.389793 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.389768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:05.408760 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.408729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9" event={"ID":"314c6992-1b3b-480a-8ccb-d2fe0e092ddd","Type":"ContainerStarted","Data":"f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4"}
Jul 09 10:39:05.408863 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.408798    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9" event={"ID":"314c6992-1b3b-480a-8ccb-d2fe0e092ddd","Type":"ContainerStarted","Data":"7c387aec854ebdecf1f9e86cba6d75c9aae2382afcec0a231f4f7635b50e1533"}
Jul 09 10:39:05.454288 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.454185    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9" podStartSLOduration=1.454166139 podStartE2EDuration="1.454166139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:39:05.45404217 +0000 UTC m=+3937.779004477" watchObservedRunningTime="2026-07-09 10:39:05.454166139 +0000 UTC m=+3937.779128425"
Jul 09 10:39:05.545576 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.545541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"]
Jul 09 10:39:05.548616 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:39:05.548581    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd766ea3f_032c_425e_9438_15d4e7bbfa05.slice/crio-76c73bfd2c5fe64d1cb5d5cc0412f651967664f69d449834779452a29c869c61 WatchSource:0}: Error finding container 76c73bfd2c5fe64d1cb5d5cc0412f651967664f69d449834779452a29c869c61: Status 404 returned error can't find the container with id 76c73bfd2c5fe64d1cb5d5cc0412f651967664f69d449834779452a29c869c61
Jul 09 10:39:05.927916 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:05.927815    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:06.414912 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:06.414860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc" event={"ID":"d766ea3f-032c-425e-9438-15d4e7bbfa05","Type":"ContainerStarted","Data":"88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc"}
Jul 09 10:39:06.415085 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:06.414928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc" event={"ID":"d766ea3f-032c-425e-9438-15d4e7bbfa05","Type":"ContainerStarted","Data":"76c73bfd2c5fe64d1cb5d5cc0412f651967664f69d449834779452a29c869c61"}
Jul 09 10:39:06.415171 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:06.415150    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:06.440025 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:06.439975    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc" podStartSLOduration=1.4399608769999999 podStartE2EDuration="1.439960877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:39:06.437739613 +0000 UTC m=+3938.762701897" watchObservedRunningTime="2026-07-09 10:39:06.439960877 +0000 UTC m=+3938.764923165"
Jul 09 10:39:14.928065 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:14.928032    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:14.940549 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:14.940523    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:15.459674 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:15.459645    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:37.421224 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:37.421191    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc" podUID="d766ea3f-032c-425e-9438-15d4e7bbfa05" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 09 10:39:43.018656 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.018614    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"]
Jul 09 10:39:43.020234 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.020197    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc" podUID="d766ea3f-032c-425e-9438-15d4e7bbfa05" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 09 10:39:43.027400 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:39:43.027368    2581 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 09 10:39:43.027400 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 10:39:43.027400 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 10:39:43.027400 ip-10-0-129-157 kubenswrapper[2581]:  > containerID="88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc" cmd=["/bin/sleep","15"]
Jul 09 10:39:43.027617 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.027414    2581 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc" message="" err=<
Jul 09 10:39:43.027617 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 10:39:43.027617 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 10:39:43.027617 ip-10-0-129-157 kubenswrapper[2581]:  >
Jul 09 10:39:43.027617 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:39:43.027425    2581 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 09 10:39:43.027617 ip-10-0-129-157 kubenswrapper[2581]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 09 10:39:43.027617 ip-10-0-129-157 kubenswrapper[2581]:         , stdout: , stderr: , exit code -1
Jul 09 10:39:43.027617 ip-10-0-129-157 kubenswrapper[2581]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc" podUID="d766ea3f-032c-425e-9438-15d4e7bbfa05" containerName="main" containerID="cri-o://88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc"
Jul 09 10:39:43.027617 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.027462    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc" podUID="d766ea3f-032c-425e-9438-15d4e7bbfa05" containerName="main" containerID="cri-o://88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc" gracePeriod=60
Jul 09 10:39:43.075879 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.075852    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"]
Jul 09 10:39:43.274486 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.274429    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:43.294965 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.294939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhk2w\" (UniqueName: \"kubernetes.io/projected/d766ea3f-032c-425e-9438-15d4e7bbfa05-kube-api-access-bhk2w\") pod \"d766ea3f-032c-425e-9438-15d4e7bbfa05\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") "
Jul 09 10:39:43.295094 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.295005    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d766ea3f-032c-425e-9438-15d4e7bbfa05-tokenizer-uds\") pod \"d766ea3f-032c-425e-9438-15d4e7bbfa05\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") "
Jul 09 10:39:43.295094 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.295035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d766ea3f-032c-425e-9438-15d4e7bbfa05-tls-certs\") pod \"d766ea3f-032c-425e-9438-15d4e7bbfa05\" (UID: \"d766ea3f-032c-425e-9438-15d4e7bbfa05\") "
Jul 09 10:39:43.295257 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.295236    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d766ea3f-032c-425e-9438-15d4e7bbfa05-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "d766ea3f-032c-425e-9438-15d4e7bbfa05" (UID: "d766ea3f-032c-425e-9438-15d4e7bbfa05"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:39:43.297118 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.297095    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d766ea3f-032c-425e-9438-15d4e7bbfa05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d766ea3f-032c-425e-9438-15d4e7bbfa05" (UID: "d766ea3f-032c-425e-9438-15d4e7bbfa05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:39:43.297118 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.297108    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d766ea3f-032c-425e-9438-15d4e7bbfa05-kube-api-access-bhk2w" (OuterVolumeSpecName: "kube-api-access-bhk2w") pod "d766ea3f-032c-425e-9438-15d4e7bbfa05" (UID: "d766ea3f-032c-425e-9438-15d4e7bbfa05"). InnerVolumeSpecName "kube-api-access-bhk2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:39:43.396457 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.396425    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d766ea3f-032c-425e-9438-15d4e7bbfa05-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:39:43.396457 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.396454    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhk2w\" (UniqueName: \"kubernetes.io/projected/d766ea3f-032c-425e-9438-15d4e7bbfa05-kube-api-access-bhk2w\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:39:43.396457 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.396465    2581 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/d766ea3f-032c-425e-9438-15d4e7bbfa05-tokenizer-uds\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:39:43.558734 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.558654    2581 generic.go:356] "Generic (PLEG): container finished" podID="d766ea3f-032c-425e-9438-15d4e7bbfa05" containerID="88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc" exitCode=0
Jul 09 10:39:43.558734 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.558700    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"
Jul 09 10:39:43.559000 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.558704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc" event={"ID":"d766ea3f-032c-425e-9438-15d4e7bbfa05","Type":"ContainerDied","Data":"88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc"}
Jul 09 10:39:43.559000 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.558829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc" event={"ID":"d766ea3f-032c-425e-9438-15d4e7bbfa05","Type":"ContainerDied","Data":"76c73bfd2c5fe64d1cb5d5cc0412f651967664f69d449834779452a29c869c61"}
Jul 09 10:39:43.559000 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.558857    2581 scope.go:117] "RemoveContainer" containerID="88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc"
Jul 09 10:39:43.567580 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.567564    2581 scope.go:117] "RemoveContainer" containerID="88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc"
Jul 09 10:39:43.567823 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:39:43.567802    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc\": container with ID starting with 88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc not found: ID does not exist" containerID="88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc"
Jul 09 10:39:43.567901 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.567830    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc"} err="failed to get container status \"88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc\": rpc error: code = NotFound desc = could not find container \"88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc\": container with ID starting with 88c034377c075b87db7729bbaf03d984e0e67a6814434f42d42538c14fa120dc not found: ID does not exist"
Jul 09 10:39:43.583841 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.583816    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"]
Jul 09 10:39:43.589181 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:43.589161    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dflgvc"]
Jul 09 10:39:44.227668 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:44.227635    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d766ea3f-032c-425e-9438-15d4e7bbfa05" path="/var/lib/kubelet/pods/d766ea3f-032c-425e-9438-15d4e7bbfa05/volumes"
Jul 09 10:39:58.095065 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.094996    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9" podUID="314c6992-1b3b-480a-8ccb-d2fe0e092ddd" containerName="main" containerID="cri-o://f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4" gracePeriod=45
Jul 09 10:39:58.340618 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.340594    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:58.410154 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.410075    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-dshm\") pod \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") "
Jul 09 10:39:58.410154 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.410109    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-model-cache\") pod \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") "
Jul 09 10:39:58.410154 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.410146    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tmp-dir\") pod \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") "
Jul 09 10:39:58.410429 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.410189    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tls-certs\") pod \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") "
Jul 09 10:39:58.410429 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.410250    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-home\") pod \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") "
Jul 09 10:39:58.410429 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.410289    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9vhv\" (UniqueName: \"kubernetes.io/projected/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-kube-api-access-n9vhv\") pod \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\" (UID: \"314c6992-1b3b-480a-8ccb-d2fe0e092ddd\") "
Jul 09 10:39:58.410574 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.410423    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "314c6992-1b3b-480a-8ccb-d2fe0e092ddd" (UID: "314c6992-1b3b-480a-8ccb-d2fe0e092ddd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:39:58.410574 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.410434    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-model-cache" (OuterVolumeSpecName: "model-cache") pod "314c6992-1b3b-480a-8ccb-d2fe0e092ddd" (UID: "314c6992-1b3b-480a-8ccb-d2fe0e092ddd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:39:58.410574 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.410510    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-home" (OuterVolumeSpecName: "home") pod "314c6992-1b3b-480a-8ccb-d2fe0e092ddd" (UID: "314c6992-1b3b-480a-8ccb-d2fe0e092ddd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:39:58.412326 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.412306    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "314c6992-1b3b-480a-8ccb-d2fe0e092ddd" (UID: "314c6992-1b3b-480a-8ccb-d2fe0e092ddd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:39:58.412406 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.412339    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-kube-api-access-n9vhv" (OuterVolumeSpecName: "kube-api-access-n9vhv") pod "314c6992-1b3b-480a-8ccb-d2fe0e092ddd" (UID: "314c6992-1b3b-480a-8ccb-d2fe0e092ddd"). InnerVolumeSpecName "kube-api-access-n9vhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:39:58.412445 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.412402    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-dshm" (OuterVolumeSpecName: "dshm") pod "314c6992-1b3b-480a-8ccb-d2fe0e092ddd" (UID: "314c6992-1b3b-480a-8ccb-d2fe0e092ddd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:39:58.511289 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.511260    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:39:58.511289 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.511287    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:39:58.511459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.511296    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9vhv\" (UniqueName: \"kubernetes.io/projected/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-kube-api-access-n9vhv\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:39:58.511459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.511306    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:39:58.511459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.511315    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:39:58.511459 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.511323    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/314c6992-1b3b-480a-8ccb-d2fe0e092ddd-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:39:58.612435 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.612400    2581 generic.go:356] "Generic (PLEG): container finished" podID="314c6992-1b3b-480a-8ccb-d2fe0e092ddd" containerID="f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4" exitCode=0
Jul 09 10:39:58.612620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.612441    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"
Jul 09 10:39:58.612620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.612487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9" event={"ID":"314c6992-1b3b-480a-8ccb-d2fe0e092ddd","Type":"ContainerDied","Data":"f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4"}
Jul 09 10:39:58.612620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.612544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9" event={"ID":"314c6992-1b3b-480a-8ccb-d2fe0e092ddd","Type":"ContainerDied","Data":"7c387aec854ebdecf1f9e86cba6d75c9aae2382afcec0a231f4f7635b50e1533"}
Jul 09 10:39:58.612620 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.612575    2581 scope.go:117] "RemoveContainer" containerID="f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4"
Jul 09 10:39:58.621014 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.620998    2581 scope.go:117] "RemoveContainer" containerID="f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4"
Jul 09 10:39:58.621264 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:39:58.621248    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4\": container with ID starting with f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4 not found: ID does not exist" containerID="f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4"
Jul 09 10:39:58.621313 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.621272    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4"} err="failed to get container status \"f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4\": rpc error: code = NotFound desc = could not find container \"f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4\": container with ID starting with f4964ded2eaaa78a5ff9d6f3e7dea88d83dafd52e5cba1ed272f140a14c47cb4 not found: ID does not exist"
Jul 09 10:39:58.635788 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.635766    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"]
Jul 09 10:39:58.642380 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:39:58.642352    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-5c5cfcf776-p2kt9"]
Jul 09 10:40:00.227543 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:00.227515    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="314c6992-1b3b-480a-8ccb-d2fe0e092ddd" path="/var/lib/kubelet/pods/314c6992-1b3b-480a-8ccb-d2fe0e092ddd/volumes"
Jul 09 10:40:03.742194 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.742157    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"]
Jul 09 10:40:03.742651 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.742433    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d766ea3f-032c-425e-9438-15d4e7bbfa05" containerName="main"
Jul 09 10:40:03.742651 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.742445    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d766ea3f-032c-425e-9438-15d4e7bbfa05" containerName="main"
Jul 09 10:40:03.742651 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.742470    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="314c6992-1b3b-480a-8ccb-d2fe0e092ddd" containerName="main"
Jul 09 10:40:03.742651 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.742475    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="314c6992-1b3b-480a-8ccb-d2fe0e092ddd" containerName="main"
Jul 09 10:40:03.742651 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.742519    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d766ea3f-032c-425e-9438-15d4e7bbfa05" containerName="main"
Jul 09 10:40:03.742651 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.742527    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="314c6992-1b3b-480a-8ccb-d2fe0e092ddd" containerName="main"
Jul 09 10:40:03.746460 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.746443    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"]
Jul 09 10:40:03.746575 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.746556    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.748999 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.748973    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 09 10:40:03.749751 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.749735    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.750210 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.750191    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 09 10:40:03.750597 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.750583    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 09 10:40:03.753839 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.753816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-9hxcn\""
Jul 09 10:40:03.762251 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.762233    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"]
Jul 09 10:40:03.763287 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.763269    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vkhz8\""
Jul 09 10:40:03.773345 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.773319    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"]
Jul 09 10:40:03.851088 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851040    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.851088 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851089    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.851303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.851303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.851303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.851303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851220    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xcv\" (UniqueName: \"kubernetes.io/projected/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-kube-api-access-d2xcv\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.851303 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.851471 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851314    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.851471 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851343    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf866fc-ab98-4dea-9404-4a9662f84321-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.851471 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pw7\" (UniqueName: \"kubernetes.io/projected/bbf866fc-ab98-4dea-9404-4a9662f84321-kube-api-access-66pw7\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.851471 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.851471 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.851413    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.952523 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.952523 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.952756 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952547    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xcv\" (UniqueName: \"kubernetes.io/projected/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-kube-api-access-d2xcv\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.952756 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.952756 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.952756 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf866fc-ab98-4dea-9404-4a9662f84321-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.952756 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pw7\" (UniqueName: \"kubernetes.io/projected/bbf866fc-ab98-4dea-9404-4a9662f84321-kube-api-access-66pw7\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.952756 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.952756 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.952756 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.953096 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.953096 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.952815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.953096 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.953007    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.953096 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.953025    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.953096 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.953006    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.953096 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.953028    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.953386 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.953307    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.953506 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.953484    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.955667 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.955638    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.955762 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.955723    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.956068 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.956047    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf866fc-ab98-4dea-9404-4a9662f84321-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:03.956116 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.956053    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.964260 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.964237    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xcv\" (UniqueName: \"kubernetes.io/projected/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-kube-api-access-d2xcv\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:03.965230 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:03.965213    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pw7\" (UniqueName: \"kubernetes.io/projected/bbf866fc-ab98-4dea-9404-4a9662f84321-kube-api-access-66pw7\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:04.057710 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:04.057619    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:04.063193 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:04.063121    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:04.209183 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:04.209159    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"]
Jul 09 10:40:04.211313 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:40:04.211287    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf866fc_ab98_4dea_9404_4a9662f84321.slice/crio-1d70f4658e2510cb097d43e7cc3b64b3236ccee60444a8aa558e3ce17e6a1b17 WatchSource:0}: Error finding container 1d70f4658e2510cb097d43e7cc3b64b3236ccee60444a8aa558e3ce17e6a1b17: Status 404 returned error can't find the container with id 1d70f4658e2510cb097d43e7cc3b64b3236ccee60444a8aa558e3ce17e6a1b17
Jul 09 10:40:04.229768 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:04.229740    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"]
Jul 09 10:40:04.232184 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:40:04.232159    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c8c8b0_01e8_4423_9b18_c6a981fc74f5.slice/crio-794b48df8d28a74edc91c5101fb654e1987f8f29b26d5eaf89f21b4851e93a29 WatchSource:0}: Error finding container 794b48df8d28a74edc91c5101fb654e1987f8f29b26d5eaf89f21b4851e93a29: Status 404 returned error can't find the container with id 794b48df8d28a74edc91c5101fb654e1987f8f29b26d5eaf89f21b4851e93a29
Jul 09 10:40:04.637707 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:04.637670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5" event={"ID":"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5","Type":"ContainerStarted","Data":"a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054"}
Jul 09 10:40:04.637937 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:04.637733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5" event={"ID":"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5","Type":"ContainerStarted","Data":"794b48df8d28a74edc91c5101fb654e1987f8f29b26d5eaf89f21b4851e93a29"}
Jul 09 10:40:04.637937 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:04.637762    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:04.639188 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:04.639162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9" event={"ID":"bbf866fc-ab98-4dea-9404-4a9662f84321","Type":"ContainerStarted","Data":"30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07"}
Jul 09 10:40:04.639273 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:04.639216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9" event={"ID":"bbf866fc-ab98-4dea-9404-4a9662f84321","Type":"ContainerStarted","Data":"1d70f4658e2510cb097d43e7cc3b64b3236ccee60444a8aa558e3ce17e6a1b17"}
Jul 09 10:40:04.698726 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:04.698682    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9" podStartSLOduration=1.6986674069999999 podStartE2EDuration="1.698667407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:40:04.695707876 +0000 UTC m=+3997.020670161" watchObservedRunningTime="2026-07-09 10:40:04.698667407 +0000 UTC m=+3997.023629690"
Jul 09 10:40:05.057976 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:05.057948    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:05.644449 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:05.644417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5" event={"ID":"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5","Type":"ContainerStarted","Data":"a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09"}
Jul 09 10:40:05.670178 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:05.670128    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5" podStartSLOduration=2.670111695 podStartE2EDuration="2.670111695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:40:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:40:05.668287351 +0000 UTC m=+3997.993249626" watchObservedRunningTime="2026-07-09 10:40:05.670111695 +0000 UTC m=+3997.995074043"
Jul 09 10:40:06.063951 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:06.063916    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:14.058949 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:14.058906    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:14.063648 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:14.063617    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:14.072395 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:14.072368    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:14.076047 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:14.076025    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:14.094449 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:14.094429    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:14.694428 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:14.694403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:14.695184 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:14.695167    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:37.701625 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:37.701591    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"]
Jul 09 10:40:37.709354 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:37.709325    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"]
Jul 09 10:40:52.721925 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:52.721809    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5" podUID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" containerName="main" containerID="cri-o://a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09" gracePeriod=45
Jul 09 10:40:52.732370 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:52.732334    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9" podUID="bbf866fc-ab98-4dea-9404-4a9662f84321" containerName="main" containerID="cri-o://30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07" gracePeriod=45
Jul 09 10:40:52.837731 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:52.837683    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5" podUID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" containerName="llm-d-routing-sidecar" containerID="cri-o://a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054" gracePeriod=45
Jul 09 10:40:52.975151 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:52.975098    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:53.048935 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.048904    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-dshm\") pod \"bbf866fc-ab98-4dea-9404-4a9662f84321\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") "
Jul 09 10:40:53.048935 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.048939    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-home\") pod \"bbf866fc-ab98-4dea-9404-4a9662f84321\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") "
Jul 09 10:40:53.049162 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.048961    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-tmp-dir\") pod \"bbf866fc-ab98-4dea-9404-4a9662f84321\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") "
Jul 09 10:40:53.049162 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.048991    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66pw7\" (UniqueName: \"kubernetes.io/projected/bbf866fc-ab98-4dea-9404-4a9662f84321-kube-api-access-66pw7\") pod \"bbf866fc-ab98-4dea-9404-4a9662f84321\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") "
Jul 09 10:40:53.049162 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.049021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf866fc-ab98-4dea-9404-4a9662f84321-tls-certs\") pod \"bbf866fc-ab98-4dea-9404-4a9662f84321\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") "
Jul 09 10:40:53.049162 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.049058    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-model-cache\") pod \"bbf866fc-ab98-4dea-9404-4a9662f84321\" (UID: \"bbf866fc-ab98-4dea-9404-4a9662f84321\") "
Jul 09 10:40:53.049391 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.049200    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-home" (OuterVolumeSpecName: "home") pod "bbf866fc-ab98-4dea-9404-4a9662f84321" (UID: "bbf866fc-ab98-4dea-9404-4a9662f84321"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:40:53.049391 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.049216    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bbf866fc-ab98-4dea-9404-4a9662f84321" (UID: "bbf866fc-ab98-4dea-9404-4a9662f84321"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:40:53.049391 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.049300    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.049552 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.049418    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-model-cache" (OuterVolumeSpecName: "model-cache") pod "bbf866fc-ab98-4dea-9404-4a9662f84321" (UID: "bbf866fc-ab98-4dea-9404-4a9662f84321"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:40:53.052224 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.052203    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-dshm" (OuterVolumeSpecName: "dshm") pod "bbf866fc-ab98-4dea-9404-4a9662f84321" (UID: "bbf866fc-ab98-4dea-9404-4a9662f84321"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:40:53.052336 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.052203    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbf866fc-ab98-4dea-9404-4a9662f84321-kube-api-access-66pw7" (OuterVolumeSpecName: "kube-api-access-66pw7") pod "bbf866fc-ab98-4dea-9404-4a9662f84321" (UID: "bbf866fc-ab98-4dea-9404-4a9662f84321"). InnerVolumeSpecName "kube-api-access-66pw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:40:53.052336 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.052243    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbf866fc-ab98-4dea-9404-4a9662f84321-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bbf866fc-ab98-4dea-9404-4a9662f84321" (UID: "bbf866fc-ab98-4dea-9404-4a9662f84321"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:40:53.077359 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.077341    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:53.150531 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150500    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2xcv\" (UniqueName: \"kubernetes.io/projected/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-kube-api-access-d2xcv\") pod \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") "
Jul 09 10:40:53.150689 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-home\") pod \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") "
Jul 09 10:40:53.150689 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150576    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tmp-dir\") pod \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") "
Jul 09 10:40:53.150689 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150606    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tls-certs\") pod \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") "
Jul 09 10:40:53.150689 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150632    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-dshm\") pod \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") "
Jul 09 10:40:53.150689 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150662    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-model-cache\") pod \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\" (UID: \"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5\") "
Jul 09 10:40:53.150987 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150771    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-home" (OuterVolumeSpecName: "home") pod "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" (UID: "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:40:53.150987 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150900    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-home\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.150987 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150919    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.150987 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150928    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.150987 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150938    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66pw7\" (UniqueName: \"kubernetes.io/projected/bbf866fc-ab98-4dea-9404-4a9662f84321-kube-api-access-66pw7\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.150987 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150947    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbf866fc-ab98-4dea-9404-4a9662f84321-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.150987 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150959    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbf866fc-ab98-4dea-9404-4a9662f84321-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.150987 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150882    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" (UID: "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:40:53.150987 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.150970    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" (UID: "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:40:53.152699 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.152676    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-dshm" (OuterVolumeSpecName: "dshm") pod "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" (UID: "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:40:53.152787 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.152730    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-kube-api-access-d2xcv" (OuterVolumeSpecName: "kube-api-access-d2xcv") pod "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" (UID: "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5"). InnerVolumeSpecName "kube-api-access-d2xcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:40:53.152787 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.152726    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" (UID: "b2c8c8b0-01e8-4423-9b18-c6a981fc74f5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 10:40:53.251417 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.251339    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tls-certs\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.251417 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.251376    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-dshm\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.251417 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.251384    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-model-cache\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.251417 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.251393    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2xcv\" (UniqueName: \"kubernetes.io/projected/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-kube-api-access-d2xcv\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.251417 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.251403    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5-tmp-dir\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:40:53.831859 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.831829    2581 generic.go:356] "Generic (PLEG): container finished" podID="bbf866fc-ab98-4dea-9404-4a9662f84321" containerID="30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07" exitCode=0
Jul 09 10:40:53.832298 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.831874    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"
Jul 09 10:40:53.832298 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.831927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9" event={"ID":"bbf866fc-ab98-4dea-9404-4a9662f84321","Type":"ContainerDied","Data":"30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07"}
Jul 09 10:40:53.832298 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.831974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9" event={"ID":"bbf866fc-ab98-4dea-9404-4a9662f84321","Type":"ContainerDied","Data":"1d70f4658e2510cb097d43e7cc3b64b3236ccee60444a8aa558e3ce17e6a1b17"}
Jul 09 10:40:53.832298 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.832006    2581 scope.go:117] "RemoveContainer" containerID="30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07"
Jul 09 10:40:53.833570 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.833550    2581 generic.go:356] "Generic (PLEG): container finished" podID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" containerID="a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09" exitCode=0
Jul 09 10:40:53.833570 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.833570    2581 generic.go:356] "Generic (PLEG): container finished" podID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" containerID="a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054" exitCode=0
Jul 09 10:40:53.833738 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.833594    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"
Jul 09 10:40:53.833738 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.833602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5" event={"ID":"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5","Type":"ContainerDied","Data":"a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09"}
Jul 09 10:40:53.833738 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.833642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5" event={"ID":"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5","Type":"ContainerDied","Data":"a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054"}
Jul 09 10:40:53.833738 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.833668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5" event={"ID":"b2c8c8b0-01e8-4423-9b18-c6a981fc74f5","Type":"ContainerDied","Data":"794b48df8d28a74edc91c5101fb654e1987f8f29b26d5eaf89f21b4851e93a29"}
Jul 09 10:40:53.841166 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.841102    2581 scope.go:117] "RemoveContainer" containerID="30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07"
Jul 09 10:40:53.841364 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:40:53.841340    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07\": container with ID starting with 30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07 not found: ID does not exist" containerID="30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07"
Jul 09 10:40:53.841466 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.841372    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07"} err="failed to get container status \"30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07\": rpc error: code = NotFound desc = could not find container \"30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07\": container with ID starting with 30a1ab31a3d6d968e8427d14e7d33199bf1c6f49577110299764653c8aa91c07 not found: ID does not exist"
Jul 09 10:40:53.841466 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.841394    2581 scope.go:117] "RemoveContainer" containerID="a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09"
Jul 09 10:40:53.850319 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.850296    2581 scope.go:117] "RemoveContainer" containerID="a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054"
Jul 09 10:40:53.857285 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.857265    2581 scope.go:117] "RemoveContainer" containerID="a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09"
Jul 09 10:40:53.857504 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:40:53.857488    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09\": container with ID starting with a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09 not found: ID does not exist" containerID="a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09"
Jul 09 10:40:53.857556 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.857511    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09"} err="failed to get container status \"a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09\": rpc error: code = NotFound desc = could not find container \"a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09\": container with ID starting with a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09 not found: ID does not exist"
Jul 09 10:40:53.857556 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.857526    2581 scope.go:117] "RemoveContainer" containerID="a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054"
Jul 09 10:40:53.857730 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:40:53.857715    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054\": container with ID starting with a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054 not found: ID does not exist" containerID="a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054"
Jul 09 10:40:53.857769 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.857733    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054"} err="failed to get container status \"a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054\": rpc error: code = NotFound desc = could not find container \"a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054\": container with ID starting with a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054 not found: ID does not exist"
Jul 09 10:40:53.857769 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.857746    2581 scope.go:117] "RemoveContainer" containerID="a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09"
Jul 09 10:40:53.857993 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.857973    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09"} err="failed to get container status \"a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09\": rpc error: code = NotFound desc = could not find container \"a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09\": container with ID starting with a55c8131a83170be5a3dd0144d152ba96d16c342228c6eeb79640c9c6677dd09 not found: ID does not exist"
Jul 09 10:40:53.858073 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.857996    2581 scope.go:117] "RemoveContainer" containerID="a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054"
Jul 09 10:40:53.858241 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.858220    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054"} err="failed to get container status \"a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054\": rpc error: code = NotFound desc = could not find container \"a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054\": container with ID starting with a0bdce23fa7463f5192310fc4915b1a09d72ff555fc3449f4251052e32fdc054 not found: ID does not exist"
Jul 09 10:40:53.864364 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.864344    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"]
Jul 09 10:40:53.873546 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.873527    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-7xfv5"]
Jul 09 10:40:53.888553 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.888525    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"]
Jul 09 10:40:53.891002 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:53.890984    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-prefill-5fc52pbt9"]
Jul 09 10:40:54.227806 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:54.227776    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" path="/var/lib/kubelet/pods/b2c8c8b0-01e8-4423-9b18-c6a981fc74f5/volumes"
Jul 09 10:40:54.228221 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:40:54.228207    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbf866fc-ab98-4dea-9404-4a9662f84321" path="/var/lib/kubelet/pods/bbf866fc-ab98-4dea-9404-4a9662f84321/volumes"
Jul 09 10:43:30.271588 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:43:30.271553    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:43:30.279280 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:43:30.279260    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:48:30.292367 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:48:30.292289    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:48:30.299942 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:48:30.299922    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:53:30.313224 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:53:30.313098    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:53:30.322367 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:53:30.322346    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:58:30.333297 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:30.333186    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:58:30.343770 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:30.343749    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vmx6b_c9e611ab-51d8-4f28-9106-829c3202216c/ovn-acl-logging/0.log"
Jul 09 10:58:35.782601 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.782569    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c4n8h/must-gather-h5xzb"]
Jul 09 10:58:35.782978 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.782850    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" containerName="llm-d-routing-sidecar"
Jul 09 10:58:35.782978 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.782862    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" containerName="llm-d-routing-sidecar"
Jul 09 10:58:35.782978 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.782875    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbf866fc-ab98-4dea-9404-4a9662f84321" containerName="main"
Jul 09 10:58:35.782978 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.782880    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbf866fc-ab98-4dea-9404-4a9662f84321" containerName="main"
Jul 09 10:58:35.782978 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.782906    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" containerName="main"
Jul 09 10:58:35.782978 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.782915    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" containerName="main"
Jul 09 10:58:35.782978 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.782967    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbf866fc-ab98-4dea-9404-4a9662f84321" containerName="main"
Jul 09 10:58:35.782978 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.782975    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" containerName="llm-d-routing-sidecar"
Jul 09 10:58:35.782978 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.782981    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c8c8b0-01e8-4423-9b18-c6a981fc74f5" containerName="main"
Jul 09 10:58:35.785715 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.785699    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:58:35.788305 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.788284    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c4n8h\"/\"kube-root-ca.crt\""
Jul 09 10:58:35.788503 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.788490    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c4n8h\"/\"openshift-service-ca.crt\""
Jul 09 10:58:35.805489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.805463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c4n8h/must-gather-h5xzb"]
Jul 09 10:58:35.923712 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.923675    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmnx7\" (UniqueName: \"kubernetes.io/projected/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-kube-api-access-qmnx7\") pod \"must-gather-h5xzb\" (UID: \"f4bbf21c-9f30-48a9-a8d1-cd383499d72c\") " pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:58:35.923918 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:35.923745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-must-gather-output\") pod \"must-gather-h5xzb\" (UID: \"f4bbf21c-9f30-48a9-a8d1-cd383499d72c\") " pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:58:36.025078 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:36.025047    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-must-gather-output\") pod \"must-gather-h5xzb\" (UID: \"f4bbf21c-9f30-48a9-a8d1-cd383499d72c\") " pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:58:36.025243 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:36.025102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmnx7\" (UniqueName: \"kubernetes.io/projected/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-kube-api-access-qmnx7\") pod \"must-gather-h5xzb\" (UID: \"f4bbf21c-9f30-48a9-a8d1-cd383499d72c\") " pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:58:36.025380 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:36.025360    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-must-gather-output\") pod \"must-gather-h5xzb\" (UID: \"f4bbf21c-9f30-48a9-a8d1-cd383499d72c\") " pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:58:36.037565 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:36.037505    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmnx7\" (UniqueName: \"kubernetes.io/projected/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-kube-api-access-qmnx7\") pod \"must-gather-h5xzb\" (UID: \"f4bbf21c-9f30-48a9-a8d1-cd383499d72c\") " pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:58:36.094167 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:36.094140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:58:36.220713 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:36.220684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c4n8h/must-gather-h5xzb"]
Jul 09 10:58:36.223659 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:58:36.223632    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4bbf21c_9f30_48a9_a8d1_cd383499d72c.slice/crio-8836f3129ebbea280e7f9628f6abe02c02e770ddea626df453669a3820752d03 WatchSource:0}: Error finding container 8836f3129ebbea280e7f9628f6abe02c02e770ddea626df453669a3820752d03: Status 404 returned error can't find the container with id 8836f3129ebbea280e7f9628f6abe02c02e770ddea626df453669a3820752d03
Jul 09 10:58:36.225627 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:36.225612    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 10:58:36.624815 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:36.624779    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c4n8h/must-gather-h5xzb" event={"ID":"f4bbf21c-9f30-48a9-a8d1-cd383499d72c","Type":"ContainerStarted","Data":"8836f3129ebbea280e7f9628f6abe02c02e770ddea626df453669a3820752d03"}
Jul 09 10:58:42.652947 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:42.652837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c4n8h/must-gather-h5xzb" event={"ID":"f4bbf21c-9f30-48a9-a8d1-cd383499d72c","Type":"ContainerStarted","Data":"a78e651b8981905a712a17809329dbbcad0181e8e2ba9d93296c17ef93d90ff0"}
Jul 09 10:58:42.652947 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:42.652881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c4n8h/must-gather-h5xzb" event={"ID":"f4bbf21c-9f30-48a9-a8d1-cd383499d72c","Type":"ContainerStarted","Data":"316dfcb6ea77121b93927d62ca315ab59665256e095d191aefcd00ef74c6b10d"}
Jul 09 10:58:42.670951 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:58:42.670877    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c4n8h/must-gather-h5xzb" podStartSLOduration=1.538513706 podStartE2EDuration="7.670861478s" totalImagesPullingTime="6.132347772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:58:42.669757941 +0000 UTC m=+5114.994720225" watchObservedRunningTime="2026-07-09 10:58:42.670861478 +0000 UTC m=+5114.995823762"
Jul 09 10:59:21.691840 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:21.691742    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:21.751121 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:21.751094    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:22.943295 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:22.943263    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:22.986487 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:22.986459    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:24.177916 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:24.177859    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:24.219489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:24.219460    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:25.372233 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:25.372201    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:25.421315 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:25.421287    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:26.580936 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:26.580903    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:26.625546 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:26.625519    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:27.806995 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:27.806965    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:27.855777 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:27.855752    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:29.030962 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:29.030928    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:29.075102 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:29.075076    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:30.257144 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:30.257112    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:30.301952 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:30.301928    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:31.511034 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:31.511002    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:31.552225 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:31.552192    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:32.721532 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:32.721471    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:32.764973 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:32.764947    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:33.929685 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:33.929658    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:33.977141 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:33.977110    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:35.869132 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:35.869101    2581 generic.go:356] "Generic (PLEG): container finished" podID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" containerID="316dfcb6ea77121b93927d62ca315ab59665256e095d191aefcd00ef74c6b10d" exitCode=0
Jul 09 10:59:35.869668 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:35.869179    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c4n8h/must-gather-h5xzb" event={"ID":"f4bbf21c-9f30-48a9-a8d1-cd383499d72c","Type":"ContainerDied","Data":"316dfcb6ea77121b93927d62ca315ab59665256e095d191aefcd00ef74c6b10d"}
Jul 09 10:59:35.869668 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:35.869563    2581 scope.go:117] "RemoveContainer" containerID="316dfcb6ea77121b93927d62ca315ab59665256e095d191aefcd00ef74c6b10d"
Jul 09 10:59:35.929776 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:35.929750    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c4n8h_must-gather-h5xzb_f4bbf21c-9f30-48a9-a8d1-cd383499d72c/gather/0.log"
Jul 09 10:59:36.793259 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:36.793225    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d8w7l/must-gather-78xgd"]
Jul 09 10:59:36.796087 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:36.796066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8w7l/must-gather-78xgd"
Jul 09 10:59:36.798600 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:36.798579    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d8w7l\"/\"kube-root-ca.crt\""
Jul 09 10:59:36.798708 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:36.798625    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d8w7l\"/\"default-dockercfg-zlpx4\""
Jul 09 10:59:36.799373 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:36.799351    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d8w7l\"/\"openshift-service-ca.crt\""
Jul 09 10:59:36.806500 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:36.806477    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8w7l/must-gather-78xgd"]
Jul 09 10:59:36.917671 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:36.917636    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z9dc\" (UniqueName: \"kubernetes.io/projected/7737840a-90c4-4d83-b88f-2ffdb1aba74d-kube-api-access-8z9dc\") pod \"must-gather-78xgd\" (UID: \"7737840a-90c4-4d83-b88f-2ffdb1aba74d\") " pod="openshift-must-gather-d8w7l/must-gather-78xgd"
Jul 09 10:59:36.918074 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:36.917683    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7737840a-90c4-4d83-b88f-2ffdb1aba74d-must-gather-output\") pod \"must-gather-78xgd\" (UID: \"7737840a-90c4-4d83-b88f-2ffdb1aba74d\") " pod="openshift-must-gather-d8w7l/must-gather-78xgd"
Jul 09 10:59:37.018982 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:37.018946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7737840a-90c4-4d83-b88f-2ffdb1aba74d-must-gather-output\") pod \"must-gather-78xgd\" (UID: \"7737840a-90c4-4d83-b88f-2ffdb1aba74d\") " pod="openshift-must-gather-d8w7l/must-gather-78xgd"
Jul 09 10:59:37.019115 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:37.019021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z9dc\" (UniqueName: \"kubernetes.io/projected/7737840a-90c4-4d83-b88f-2ffdb1aba74d-kube-api-access-8z9dc\") pod \"must-gather-78xgd\" (UID: \"7737840a-90c4-4d83-b88f-2ffdb1aba74d\") " pod="openshift-must-gather-d8w7l/must-gather-78xgd"
Jul 09 10:59:37.019314 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:37.019296    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7737840a-90c4-4d83-b88f-2ffdb1aba74d-must-gather-output\") pod \"must-gather-78xgd\" (UID: \"7737840a-90c4-4d83-b88f-2ffdb1aba74d\") " pod="openshift-must-gather-d8w7l/must-gather-78xgd"
Jul 09 10:59:37.027628 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:37.027600    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z9dc\" (UniqueName: \"kubernetes.io/projected/7737840a-90c4-4d83-b88f-2ffdb1aba74d-kube-api-access-8z9dc\") pod \"must-gather-78xgd\" (UID: \"7737840a-90c4-4d83-b88f-2ffdb1aba74d\") " pod="openshift-must-gather-d8w7l/must-gather-78xgd"
Jul 09 10:59:37.105308 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:37.105214    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8w7l/must-gather-78xgd"
Jul 09 10:59:37.240133 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:37.240102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8w7l/must-gather-78xgd"]
Jul 09 10:59:37.243193 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:59:37.243168    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7737840a_90c4_4d83_b88f_2ffdb1aba74d.slice/crio-0f30c0a314b8b086c2d345cb36190fef4b602b0c3de03a892034b0239fdea92b WatchSource:0}: Error finding container 0f30c0a314b8b086c2d345cb36190fef4b602b0c3de03a892034b0239fdea92b: Status 404 returned error can't find the container with id 0f30c0a314b8b086c2d345cb36190fef4b602b0c3de03a892034b0239fdea92b
Jul 09 10:59:37.877693 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:37.877660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8w7l/must-gather-78xgd" event={"ID":"7737840a-90c4-4d83-b88f-2ffdb1aba74d","Type":"ContainerStarted","Data":"0f30c0a314b8b086c2d345cb36190fef4b602b0c3de03a892034b0239fdea92b"}
Jul 09 10:59:38.886059 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:38.885313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8w7l/must-gather-78xgd" event={"ID":"7737840a-90c4-4d83-b88f-2ffdb1aba74d","Type":"ContainerStarted","Data":"d6fbf7eab3d523218b1f8da2eee1a22fff9b6405eef6be5a7c5cc9dff7acdb06"}
Jul 09 10:59:38.886059 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:38.885365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8w7l/must-gather-78xgd" event={"ID":"7737840a-90c4-4d83-b88f-2ffdb1aba74d","Type":"ContainerStarted","Data":"d12c19a1c4cb7b3d24edd9ac5e8942c88d9b816b803a6f4cf5e9ec8873a2a8b4"}
Jul 09 10:59:38.902119 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:38.902065    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d8w7l/must-gather-78xgd" podStartSLOduration=2.105675137 podStartE2EDuration="2.902043275s" totalImagesPullingTime="796.368138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 10:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:59:38.901840081 +0000 UTC m=+5171.226802365" watchObservedRunningTime="2026-07-09 10:59:38.902043275 +0000 UTC m=+5171.227005559"
Jul 09 10:59:39.596489 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:39.596459    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wg7tt_8aa5ea73-c624-40b0-8672-e1b09f0ee72b/global-pull-secret-syncer/0.log"
Jul 09 10:59:39.746539 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:39.746506    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x6tpp_4efaeb6b-b220-4e84-ac19-115ce6ef37e8/konnectivity-agent/0.log"
Jul 09 10:59:39.803419 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:39.803381    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-157.ec2.internal_78255babc9a1b932c1400b78f8b89518/haproxy/0.log"
Jul 09 10:59:41.626246 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:41.626203    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-c4n8h/must-gather-h5xzb"]
Jul 09 10:59:41.626703 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:41.626433    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-c4n8h/must-gather-h5xzb" podUID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" containerName="copy" containerID="cri-o://a78e651b8981905a712a17809329dbbcad0181e8e2ba9d93296c17ef93d90ff0" gracePeriod=2
Jul 09 10:59:41.631511 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:41.631482    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-c4n8h/must-gather-h5xzb"]
Jul 09 10:59:41.909264 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:41.909118    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c4n8h_must-gather-h5xzb_f4bbf21c-9f30-48a9-a8d1-cd383499d72c/copy/0.log"
Jul 09 10:59:41.909841 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:41.909748    2581 generic.go:356] "Generic (PLEG): container finished" podID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" containerID="a78e651b8981905a712a17809329dbbcad0181e8e2ba9d93296c17ef93d90ff0" exitCode=143
Jul 09 10:59:42.006909 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.005652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c4n8h_must-gather-h5xzb_f4bbf21c-9f30-48a9-a8d1-cd383499d72c/copy/0.log"
Jul 09 10:59:42.010913 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.008995    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:59:42.012140 ip-10-0-129-157 kubenswrapper[2581]: E0709 10:59:42.011120    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-h5xzb\" is forbidden: User \"system:node:ip-10-0-129-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-c4n8h\": no relationship found between node 'ip-10-0-129-157.ec2.internal' and this object" podUID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:59:42.162970 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.161853    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-must-gather-output\") pod \"f4bbf21c-9f30-48a9-a8d1-cd383499d72c\" (UID: \"f4bbf21c-9f30-48a9-a8d1-cd383499d72c\") "
Jul 09 10:59:42.163658 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.163119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmnx7\" (UniqueName: \"kubernetes.io/projected/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-kube-api-access-qmnx7\") pod \"f4bbf21c-9f30-48a9-a8d1-cd383499d72c\" (UID: \"f4bbf21c-9f30-48a9-a8d1-cd383499d72c\") "
Jul 09 10:59:42.166150 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.166118    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-kube-api-access-qmnx7" (OuterVolumeSpecName: "kube-api-access-qmnx7") pod "f4bbf21c-9f30-48a9-a8d1-cd383499d72c" (UID: "f4bbf21c-9f30-48a9-a8d1-cd383499d72c"). InnerVolumeSpecName "kube-api-access-qmnx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 10:59:42.174774 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.174739    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "f4bbf21c-9f30-48a9-a8d1-cd383499d72c" (UID: "f4bbf21c-9f30-48a9-a8d1-cd383499d72c"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 10:59:42.231724 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.231680    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" path="/var/lib/kubelet/pods/f4bbf21c-9f30-48a9-a8d1-cd383499d72c/volumes"
Jul 09 10:59:42.264950 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.264899    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmnx7\" (UniqueName: \"kubernetes.io/projected/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-kube-api-access-qmnx7\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:59:42.264950 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.264942    2581 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f4bbf21c-9f30-48a9-a8d1-cd383499d72c-must-gather-output\") on node \"ip-10-0-129-157.ec2.internal\" DevicePath \"\""
Jul 09 10:59:42.917581 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.917486    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-c4n8h_must-gather-h5xzb_f4bbf21c-9f30-48a9-a8d1-cd383499d72c/copy/0.log"
Jul 09 10:59:42.918118 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.918027    2581 scope.go:117] "RemoveContainer" containerID="a78e651b8981905a712a17809329dbbcad0181e8e2ba9d93296c17ef93d90ff0"
Jul 09 10:59:42.918205 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.918182    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c4n8h/must-gather-h5xzb"
Jul 09 10:59:42.951695 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:42.949262    2581 scope.go:117] "RemoveContainer" containerID="316dfcb6ea77121b93927d62ca315ab59665256e095d191aefcd00ef74c6b10d"
Jul 09 10:59:44.972867 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:44.972762    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-z7vd9_b41afe9d-b625-4729-823d-52f537c3540c/authorino/0.log"
Jul 09 10:59:45.095504 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:45.095471    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-2ndhv_033f7d4b-49b8-41aa-a7d4-d9a9a5a59565/kuadrant-console-plugin/0.log"
Jul 09 10:59:46.821959 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:46.821636    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blcb8_a36296e6-f354-44a5-9ee7-20a9ddff8380/node-exporter/0.log"
Jul 09 10:59:46.856081 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:46.855903    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blcb8_a36296e6-f354-44a5-9ee7-20a9ddff8380/kube-rbac-proxy/0.log"
Jul 09 10:59:46.915916 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:46.911922    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-blcb8_a36296e6-f354-44a5-9ee7-20a9ddff8380/init-textfile/0.log"
Jul 09 10:59:49.526953 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.526914    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"]
Jul 09 10:59:49.527525 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.527309    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" containerName="gather"
Jul 09 10:59:49.527525 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.527327    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" containerName="gather"
Jul 09 10:59:49.527525 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.527354    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" containerName="copy"
Jul 09 10:59:49.527525 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.527363    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" containerName="copy"
Jul 09 10:59:49.527525 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.527440    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" containerName="copy"
Jul 09 10:59:49.527525 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.527450    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4bbf21c-9f30-48a9-a8d1-cd383499d72c" containerName="gather"
Jul 09 10:59:49.530770 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.530738    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.541528 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.541505    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"]
Jul 09 10:59:49.631071 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.631036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-lib-modules\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.631257 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.631090    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr4np\" (UniqueName: \"kubernetes.io/projected/46836293-f119-4d13-bea1-9df6b1fae7a6-kube-api-access-sr4np\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.631257 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.631140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-sys\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.631257 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.631173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-podres\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.631257 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.631211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-proc\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.732396 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.732356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-lib-modules\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.732602 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.732410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr4np\" (UniqueName: \"kubernetes.io/projected/46836293-f119-4d13-bea1-9df6b1fae7a6-kube-api-access-sr4np\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.732602 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.732461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-sys\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.732602 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.732494    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-podres\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.732602 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.732533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-proc\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.732602 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.732541    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-lib-modules\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.732830 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.732619    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-proc\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.732830 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.732618    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-sys\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.732830 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.732678    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/46836293-f119-4d13-bea1-9df6b1fae7a6-podres\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.742642 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.742612    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr4np\" (UniqueName: \"kubernetes.io/projected/46836293-f119-4d13-bea1-9df6b1fae7a6-kube-api-access-sr4np\") pod \"perf-node-gather-daemonset-b4m7p\" (UID: \"46836293-f119-4d13-bea1-9df6b1fae7a6\") " pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:49.844429 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:49.844342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:50.003996 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:50.003811    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"]
Jul 09 10:59:50.007553 ip-10-0-129-157 kubenswrapper[2581]: W0709 10:59:50.007526    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod46836293_f119_4d13_bea1_9df6b1fae7a6.slice/crio-b267320e84fa6dc3b00fe10ab06d98e547a6f25e615d1269c5fd0dfbf9940ed0 WatchSource:0}: Error finding container b267320e84fa6dc3b00fe10ab06d98e547a6f25e615d1269c5fd0dfbf9940ed0: Status 404 returned error can't find the container with id b267320e84fa6dc3b00fe10ab06d98e547a6f25e615d1269c5fd0dfbf9940ed0
Jul 09 10:59:50.962643 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:50.962605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p" event={"ID":"46836293-f119-4d13-bea1-9df6b1fae7a6","Type":"ContainerStarted","Data":"bfea8068e3057221ddcdd872ede68e6344cd5f5dab12993254b947273e8c2b4b"}
Jul 09 10:59:50.963206 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:50.962667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p" event={"ID":"46836293-f119-4d13-bea1-9df6b1fae7a6","Type":"ContainerStarted","Data":"b267320e84fa6dc3b00fe10ab06d98e547a6f25e615d1269c5fd0dfbf9940ed0"}
Jul 09 10:59:50.963206 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:50.962707    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:50.981226 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:50.981160    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p" podStartSLOduration=1.981138302 podStartE2EDuration="1.981138302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 10:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 10:59:50.978792036 +0000 UTC m=+5183.303754321" watchObservedRunningTime="2026-07-09 10:59:50.981138302 +0000 UTC m=+5183.306100588"
Jul 09 10:59:51.609149 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:51.609125    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fpvfk_9ffe7afe-800e-4a88-a39d-2b9bd0433a87/dns/0.log"
Jul 09 10:59:51.631916 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:51.631878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fpvfk_9ffe7afe-800e-4a88-a39d-2b9bd0433a87/kube-rbac-proxy/0.log"
Jul 09 10:59:51.731092 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:51.731061    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pk5f2_f832bbb8-9acb-4a0b-bee3-77a5d5aad1d4/dns-node-resolver/0.log"
Jul 09 10:59:52.278862 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:52.278828    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-766c776c56-ctjz7_6b7d7719-4327-4382-879d-6def3a0bcebb/registry/0.log"
Jul 09 10:59:52.302301 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:52.302269    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2fd5t_8eb0783e-86d2-42bb-96e8-c69c1fae5629/node-ca/0.log"
Jul 09 10:59:54.315128 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:54.315105    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-whkx9_bec59435-ae5d-4001-a28c-4b475d5559d0/serve-healthcheck-canary/0.log"
Jul 09 10:59:54.864133 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:54.864080    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sn8f2_9c912314-2d73-4cdb-91d1-f1f7eb547f53/kube-rbac-proxy/0.log"
Jul 09 10:59:54.886464 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:54.886437    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sn8f2_9c912314-2d73-4cdb-91d1-f1f7eb547f53/exporter/0.log"
Jul 09 10:59:54.908657 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:54.908600    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sn8f2_9c912314-2d73-4cdb-91d1-f1f7eb547f53/extractor/0.log"
Jul 09 10:59:56.979703 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:56.979056    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d8w7l/perf-node-gather-daemonset-b4m7p"
Jul 09 10:59:57.751306 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:57.751277    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-599d46c4cb-m6gx7_01877ad4-3095-4dc1-b9ab-9ba946aa691c/manager/0.log"
Jul 09 10:59:58.625434 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:58.625372    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-df5969cfc-jk8w2_8187848a-035e-4601-9d07-7c242c78d4bc/manager/0.log"
Jul 09 10:59:58.649170 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:58.649140    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-wh8l4_200a5fc6-cbde-42b2-8303-8f6f1971226e/server/0.log"
Jul 09 10:59:59.144613 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:59.144581    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hq5mf_898f57ab-45fe-45c4-bb47-3d72b3920492/s3-init/0.log"
Jul 09 10:59:59.163865 ip-10-0-129-157 kubenswrapper[2581]: I0709 10:59:59.163838    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hq5mf_898f57ab-45fe-45c4-bb47-3d72b3920492/download-hf-model/0.log"
Jul 09 11:00:04.385214 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:04.385181    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-kw85c_deaaf287-25fd-457b-95ed-1cc34da4a9d1/migrator/0.log"
Jul 09 11:00:04.416948 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:04.416918    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-kw85c_deaaf287-25fd-457b-95ed-1cc34da4a9d1/graceful-termination/0.log"
Jul 09 11:00:05.782815 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:05.782790    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tnhw_7320fb87-f8d7-4d0c-9e3a-5da86b108b62/kube-multus-additional-cni-plugins/0.log"
Jul 09 11:00:05.807676 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:05.807651    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tnhw_7320fb87-f8d7-4d0c-9e3a-5da86b108b62/egress-router-binary-copy/0.log"
Jul 09 11:00:05.829761 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:05.829731    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tnhw_7320fb87-f8d7-4d0c-9e3a-5da86b108b62/cni-plugins/0.log"
Jul 09 11:00:05.855219 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:05.855191    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tnhw_7320fb87-f8d7-4d0c-9e3a-5da86b108b62/bond-cni-plugin/0.log"
Jul 09 11:00:05.876042 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:05.876013    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tnhw_7320fb87-f8d7-4d0c-9e3a-5da86b108b62/routeoverride-cni/0.log"
Jul 09 11:00:05.899278 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:05.899249    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tnhw_7320fb87-f8d7-4d0c-9e3a-5da86b108b62/whereabouts-cni-bincopy/0.log"
Jul 09 11:00:05.921567 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:05.921543    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2tnhw_7320fb87-f8d7-4d0c-9e3a-5da86b108b62/whereabouts-cni/0.log"
Jul 09 11:00:06.320432 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:06.320406    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fxrwr_21a09ec5-5646-45b9-bcce-9c96d2e9537c/kube-multus/0.log"
Jul 09 11:00:06.341251 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:06.341223    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5g56w_cc58490a-3112-469d-b967-06c5bd8c9e20/network-metrics-daemon/0.log"
Jul 09 11:00:06.359732 ip-10-0-129-157 kubenswrapper[2581]: I0709 11:00:06.359710    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5g56w_cc58490a-3112-469d-b967-06c5bd8c9e20/kube-rbac-proxy/0.log"