Jul 07 02:15:52.731441 ip-10-0-132-101 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 07 02:15:52.731453 ip-10-0-132-101 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 07 02:15:52.731460 ip-10-0-132-101 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 07 02:15:52.731699 ip-10-0-132-101 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 07 02:16:02.928060 ip-10-0-132-101 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 07 02:16:02.928077 ip-10-0-132-101 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a763398779964ce382581a4ff4988600 --
Jul 07 02:18:06.125233 ip-10-0-132-101 systemd[1]: Starting Kubernetes Kubelet...
Jul 07 02:18:06.603469 ip-10-0-132-101 kubenswrapper[2543]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:18:06.603469 ip-10-0-132-101 kubenswrapper[2543]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 07 02:18:06.603469 ip-10-0-132-101 kubenswrapper[2543]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:18:06.603469 ip-10-0-132-101 kubenswrapper[2543]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 07 02:18:06.603469 ip-10-0-132-101 kubenswrapper[2543]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 07 02:18:06.605888 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.605800    2543 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 07 02:18:06.609440 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609417    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:06.609440 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609435    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:06.609440 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609439    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:06.609440 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609443    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:06.609440 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609447    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609451    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609455    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609459    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609462    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609467    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609471    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609485    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609490    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609493    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609497    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609501    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609505    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609509    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609512    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609516    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609523    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609529    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609534    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:06.609775 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609538    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609542    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609547    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609551    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609555    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609559    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609564    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609568    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609573    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609577    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609580    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609585    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609589    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609593    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609598    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609602    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609606    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609611    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609618    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:06.610564 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609625    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609630    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609635    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609639    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609644    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609649    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609653    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609657    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609664    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609668    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609672    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609677    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609681    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609685    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609689    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609693    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609697    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609701    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609705    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:06.611363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609710    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609716    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609722    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609727    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609731    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609736    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609741    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609744    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609751    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609757    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609762    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609766    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609779    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609784    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609788    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609792    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609796    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609800    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609804    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609808    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:06.612161 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609813    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609817    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609821    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609825    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609829    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609833    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609837    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609841    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609845    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609849    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609853    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609857    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609861    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609866    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609870    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609873    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609878    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.609882    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611101    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611111    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:06.613035 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611115    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611120    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611126    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611134    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611140    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611145    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611150    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611154    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611159    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611164    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611168    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611173    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611177    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611181    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611186    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611191    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611196    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611201    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611205    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:06.613862 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611210    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611214    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611219    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611223    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611227    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611231    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611235    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611239    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611243    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611247    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611251    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611255    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611260    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611264    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611269    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611273    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611278    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611282    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611286    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611291    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:06.614422 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611295    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611299    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611303    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611307    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611311    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611315    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611319    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611323    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611327    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611332    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611337    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611342    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611348    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611353    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611357    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611361    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611364    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611368    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611372    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:06.614903 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611376    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611380    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611384    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611389    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611393    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611398    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611402    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611407    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611411    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611416    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611421    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611426    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611430    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611434    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611438    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611442    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611447    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611451    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611456    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:06.615396 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611465    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611470    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611475    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611479    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611483    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611488    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611493    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611497    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611502    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611506    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611510    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611513    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611517    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611522    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611526    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611530    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611535    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611539    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611543    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.611547    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:06.615854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612152    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612160    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612166    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612171    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612176    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612181    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612185    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612190    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612194    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612198    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612203    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612207    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612211    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612215    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612220    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612224    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612228    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612233    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612237    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:06.616350 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612241    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612246    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612250    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612254    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612260    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612266    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612270    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612274    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612278    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612286    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612290    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612294    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612298    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612303    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612307    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612311    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612315    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612320    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612324    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:06.616918 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612329    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612333    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612337    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612341    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612345    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612349    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612353    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612357    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612361    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612366    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612370    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612374    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612378    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612382    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612386    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612390    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612395    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612399    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612404    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612408    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:06.617402 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612415    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612420    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612427    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612431    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612436    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612440    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612444    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612449    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612453    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612460    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612465    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612470    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612477    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612481    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612486    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612490    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612495    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612499    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612504    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:06.617882 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612509    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612513    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612518    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612522    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612526    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612530    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612534    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612538    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612544    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612549    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612553    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612557    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612561    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612565    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612569    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612578    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612583    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612587    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612591    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612595    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:06.618363 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612599    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.612604    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612707    2543 flags.go:64] FLAG: --address="0.0.0.0"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612728    2543 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612739    2543 flags.go:64] FLAG: --anonymous-auth="true"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612746    2543 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612753    2543 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612757    2543 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612764    2543 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612770    2543 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612775    2543 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612780    2543 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612785    2543 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612790    2543 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612795    2543 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612800    2543 flags.go:64] FLAG: --cgroup-root=""
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612804    2543 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612809    2543 flags.go:64] FLAG: --client-ca-file=""
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612813    2543 flags.go:64] FLAG: --cloud-provider="external"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612818    2543 flags.go:64] FLAG: --cluster-dns="[]"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612826    2543 flags.go:64] FLAG: --cluster-domain=""
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612831    2543 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612836    2543 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612840    2543 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 07 02:18:06.618849 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612845    2543 flags.go:64] FLAG: --container-log-max-files="5"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612851    2543 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612856    2543 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612863    2543 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612868    2543 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612873    2543 flags.go:64] FLAG: --contention-profiling="false"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612877    2543 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612882    2543 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612887    2543 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612892    2543 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612899    2543 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612904    2543 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612908    2543 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612913    2543 flags.go:64] FLAG: --enable-load-reader="false"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612918    2543 flags.go:64] FLAG: --enable-server="true"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612923    2543 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612930    2543 flags.go:64] FLAG: --event-burst="100"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612934    2543 flags.go:64] FLAG: --event-qps="50"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612939    2543 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612944    2543 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612948    2543 flags.go:64] FLAG: --eviction-hard=""
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612954    2543 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612959    2543 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612981    2543 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612986    2543 flags.go:64] FLAG: --eviction-soft=""
Jul 07 02:18:06.619481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612991    2543 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612995    2543 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.612999    2543 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613006    2543 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613011    2543 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613016    2543 flags.go:64] FLAG: --fail-swap-on="true"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613021    2543 flags.go:64] FLAG: --feature-gates=""
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613027    2543 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613031    2543 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613036    2543 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613043    2543 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613047    2543 flags.go:64] FLAG: --healthz-port="10248"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613052    2543 flags.go:64] FLAG: --help="false"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613056    2543 flags.go:64] FLAG: --hostname-override="ip-10-0-132-101.ec2.internal"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613061    2543 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613066    2543 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613071    2543 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613077    2543 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613082    2543 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613087    2543 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613092    2543 flags.go:64] FLAG: --image-service-endpoint=""
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613096    2543 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613102    2543 flags.go:64] FLAG: --kube-api-burst="100"
Jul 07 02:18:06.620086 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613107    2543 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613112    2543 flags.go:64] FLAG: --kube-api-qps="50"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613117    2543 flags.go:64] FLAG: --kube-reserved=""
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613122    2543 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613126    2543 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613131    2543 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613135    2543 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613140    2543 flags.go:64] FLAG: --lock-file=""
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613144    2543 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613149    2543 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613153    2543 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613162    2543 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613168    2543 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613173    2543 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613178    2543 flags.go:64] FLAG: --logging-format="text"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613182    2543 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613187    2543 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613192    2543 flags.go:64] FLAG: --manifest-url=""
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613196    2543 flags.go:64] FLAG: --manifest-url-header=""
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613204    2543 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613208    2543 flags.go:64] FLAG: --max-open-files="1000000"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613214    2543 flags.go:64] FLAG: --max-pods="110"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613219    2543 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613223    2543 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613228    2543 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 07 02:18:06.620817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613232    2543 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613237    2543 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613241    2543 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613246    2543 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613257    2543 flags.go:64] FLAG: --node-status-max-images="50"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613261    2543 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613266    2543 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613271    2543 flags.go:64] FLAG: --pod-cidr=""
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613276    2543 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613284    2543 flags.go:64] FLAG: --pod-manifest-path=""
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613288    2543 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613293    2543 flags.go:64] FLAG: --pods-per-core="0"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613297    2543 flags.go:64] FLAG: --port="10250"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613302    2543 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613307    2543 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bff0de61a3964184"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613312    2543 flags.go:64] FLAG: --qos-reserved=""
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613317    2543 flags.go:64] FLAG: --read-only-port="10255"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613321    2543 flags.go:64] FLAG: --register-node="true"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613326    2543 flags.go:64] FLAG: --register-with-taints=""
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613333    2543 flags.go:64] FLAG: --registry-burst="10"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613338    2543 flags.go:64] FLAG: --registry-qps="5"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613342    2543 flags.go:64] FLAG: --reserved-cpus=""
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613346    2543 flags.go:64] FLAG: --reserved-memory=""
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613352    2543 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613357    2543 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 07 02:18:06.621820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613362    2543 flags.go:64] FLAG: --rotate-certificates="false"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613367    2543 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613372    2543 flags.go:64] FLAG: --runonce="false"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613377    2543 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613381    2543 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613386    2543 flags.go:64] FLAG: --seccomp-default="false"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613391    2543 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613395    2543 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613400    2543 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613405    2543 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613410    2543 flags.go:64] FLAG: --storage-driver-password="root"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613415    2543 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613420    2543 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613424    2543 flags.go:64] FLAG: --storage-driver-user="root"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613429    2543 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613434    2543 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613439    2543 flags.go:64] FLAG: --system-cgroups=""
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613444    2543 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613452    2543 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613456    2543 flags.go:64] FLAG: --tls-cert-file=""
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613461    2543 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613468    2543 flags.go:64] FLAG: --tls-min-version=""
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613472    2543 flags.go:64] FLAG: --tls-private-key-file=""
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613477    2543 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613482    2543 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 07 02:18:06.622473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613486    2543 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613493    2543 flags.go:64] FLAG: --v="2"
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613499    2543 flags.go:64] FLAG: --version="false"
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613505    2543 flags.go:64] FLAG: --vmodule=""
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613510    2543 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.613516    2543 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613703    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613709    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613714    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613719    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613723    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613727    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613731    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613736    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613740    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613744    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613749    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613753    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613757    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613761    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613765    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:06.623110 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613769    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613773    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613778    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613782    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613786    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613790    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613795    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613799    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613803    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613807    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613811    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613815    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613821    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613826    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613830    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613834    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613838    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613843    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613847    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613852    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:06.623654 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613856    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613862    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613867    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613871    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613876    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613879    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613883    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613887    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613891    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613898    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613903    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613908    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613938    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613953    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613957    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613960    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613982    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613988    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613990    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613993    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:06.624160 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.613996    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614000    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614003    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614006    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614009    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614011    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614014    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614017    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614019    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614022    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614025    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614028    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614031    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614033    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614036    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614038    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614041    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614047    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614049    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614052    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:06.624637 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614055    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614057    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614060    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614062    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614065    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614068    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614070    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614075    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614079    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614082    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614085    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614088    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614091    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614093    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614096    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614099    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614101    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614103    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614106    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614109    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:06.625145 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614112    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614114    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614117    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.614119    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.614769    2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.622859    2543 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.622876    2543 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.623605    2543 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.623613    2543 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623717    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623723    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623727    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623730    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623734    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623738    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:06.625621 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623741    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623744    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623747    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623750    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623752    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623755    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623759    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623763    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623766    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623769    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623772    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623775    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623777    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623781    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623783    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623786    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623789    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623791    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623794    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:06.626018 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623796    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623799    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623801    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623803    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623806    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623808    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623811    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623814    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623816    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623819    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623822    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623824    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623827    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623829    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623832    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623834    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623837    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623840    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623842    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623845    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:06.626500 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623848    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623851    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623853    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623856    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623858    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623861    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623863    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623866    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623868    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623870    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623873    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623875    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623877    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623880    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623882    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623885    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623887    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623889    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623892    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:06.627091 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623895    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623897    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623900    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623902    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623905    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623907    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623910    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623912    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623914    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623918    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623920    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623923    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623926    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623928    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623930    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623939    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623942    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623946    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623949    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623952    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:06.627544 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623954    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623957    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623976    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623981    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623985    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623989    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623994    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623996    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.623999    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624001    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624004    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624006    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624009    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624011    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624014    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:06.628033 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.624019    2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624130    2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624135    2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624139    2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624142    2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624145    2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624148    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624151    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624153    2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624156    2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624158    2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624161    2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624163    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624167    2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624170    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624173    2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624175    2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624178    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624180    2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 07 02:18:06.628398 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624183    2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624185    2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624187    2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624190    2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624192    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624195    2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624197    2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624199    2543 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624202    2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624204    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624207    2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624209    2543 feature_gate.go:328] unrecognized feature gate: Example2
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624212    2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624215    2543 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624217    2543 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624220    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624222    2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624224    2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624227    2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624229    2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 07 02:18:06.628856 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624232    2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624234    2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624236    2543 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624239    2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624241    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624244    2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624246    2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624249    2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624252    2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624254    2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624257    2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624259    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624262    2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624264    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624266    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624269    2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624272    2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624274    2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624276    2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624279    2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 07 02:18:06.629357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624282    2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624285    2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624287    2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624291    2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624294    2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624296    2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624299    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624301    2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624304    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624306    2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624308    2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624311    2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624313    2543 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624316    2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624318    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624321    2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624323    2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624326    2543 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624328    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 07 02:18:06.629843 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624330    2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624333    2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624336    2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624338    2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624341    2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624343    2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624346    2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624348    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624351    2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624353    2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624355    2543 feature_gate.go:328] unrecognized feature gate: Example
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624357    2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624360    2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624362    2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624365    2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624367    2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624370    2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624372    2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624374    2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624377    2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 07 02:18:06.630324 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624379    2543 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 07 02:18:06.630801 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:06.624382    2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 07 02:18:06.630801 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.624386    2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 07 02:18:06.630801 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.624494    2543 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 07 02:18:06.630801 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.626738    2543 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 07 02:18:06.630801 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.628622    2543 server.go:1019] "Starting client certificate rotation"
Jul 07 02:18:06.630801 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.628730    2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 02:18:06.630801 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.629695    2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 07 02:18:06.631857 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.631844    2543 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 02:18:06.632999 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.632983    2543 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 07 02:18:06.644565 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.644551    2543 log.go:25] "Validated CRI v1 runtime API"
Jul 07 02:18:06.647045 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.647031    2543 log.go:25] "Validated CRI v1 image API"
Jul 07 02:18:06.647876 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.647858    2543 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 07 02:18:06.652430 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.652406    2543 fs.go:135] Filesystem UUIDs: map[459e0d1f-1f56-4fe2-86ab-3cafaf330718:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b8436123-17f5-4394-aa03-ce1ba88d1da0:/dev/nvme0n1p4]
Jul 07 02:18:06.652500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.652430    2543 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 07 02:18:06.659796 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.659677    2543 manager.go:216] Machine: {Timestamp:2026-07-07 02:18:06.657745923 +0000 UTC m=+0.401127722 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3134047 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21feb3fdf83cb2e196a2d9277e72cd SystemUUID:ec21feb3-fdf8-3cb2-e196-a2d9277e72cd BootID:a7633987-7996-4ce3-8258-1a4ff4988600 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:31:7c:53:85:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:31:7c:53:85:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:9d:97:b2:1f:24 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 07 02:18:06.659796 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.659786    2543 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 07 02:18:06.659925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.659865    2543 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 07 02:18:06.663283 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.663249    2543 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 07 02:18:06.663462 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.663447    2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 02:18:06.663515 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.663283    2543 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-101.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 07 02:18:06.664728 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.664717    2543 topology_manager.go:138] "Creating topology manager with none policy"
Jul 07 02:18:06.664766 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.664732    2543 container_manager_linux.go:309] "Creating device plugin manager"
Jul 07 02:18:06.664766 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.664746    2543 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 02:18:06.664766 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.664760    2543 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 07 02:18:06.665950 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.665938    2543 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 07 02:18:06.668913 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.668901    2543 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 02:18:06.669055 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.669045    2543 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 07 02:18:06.669095 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.669076    2543 kubelet.go:486] "Attempting to sync node with API server"
Jul 07 02:18:06.669095 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.669086    2543 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 07 02:18:06.669169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.669097    2543 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 07 02:18:06.669169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.669107    2543 kubelet.go:398] "Adding apiserver pod source"
Jul 07 02:18:06.669169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.669116    2543 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 07 02:18:06.669499 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.669488    2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 02:18:06.669535 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.669509    2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 07 02:18:06.672452 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.672436    2543 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 07 02:18:06.674565 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.674551    2543 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 07 02:18:06.674623 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.674582    2543 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 07 02:18:06.675484 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675472    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 07 02:18:06.675517 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675493    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 07 02:18:06.675517 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675500    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 07 02:18:06.675517 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675505    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 07 02:18:06.675517 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675515    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 07 02:18:06.675629 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675520    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 07 02:18:06.675629 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675528    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 07 02:18:06.675629 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675533    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 07 02:18:06.675629 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675541    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 07 02:18:06.675629 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675551    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 07 02:18:06.675629 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675569    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 07 02:18:06.675629 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.675574    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 07 02:18:06.677432 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.677419    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 07 02:18:06.677432 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.677431    2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 07 02:18:06.680771 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.680757    2543 server.go:1268] "Started kubelet"
Jul 07 02:18:06.680895 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.680869    2543 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 07 02:18:06.681742 ip-10-0-132-101 systemd[1]: Started Kubernetes Kubelet.
Jul 07 02:18:06.681877 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.681737    2543 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-101.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 07 02:18:06.681877 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.681757    2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 07 02:18:06.681877 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.681802    2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 07 02:18:06.682070 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.681997    2543 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 07 02:18:06.682103 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.682088    2543 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 07 02:18:06.683457 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.683441    2543 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 07 02:18:06.683584 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.683572    2543 server.go:310] "Adding debug handlers to kubelet server"
Jul 07 02:18:06.686264 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.686239    2543 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 07 02:18:06.688611 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.688597    2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c7f9l"
Jul 07 02:18:06.689499 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.688088    2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-101.ec2.internal.18bfe06177583af2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-101.ec2.internal,UID:ip-10-0-132-101.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-101.ec2.internal,},FirstTimestamp:2026-07-07 02:18:06.680726258 +0000 UTC m=+0.424108054,LastTimestamp:2026-07-07 02:18:06.680726258 +0000 UTC m=+0.424108054,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-101.ec2.internal,}"
Jul 07 02:18:06.689703 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.689684    2543 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 07 02:18:06.689703 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.689692    2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 07 02:18:06.689835 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.689758    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:06.690399 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.690370    2543 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 07 02:18:06.690399 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.690373    2543 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 07 02:18:06.690399 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.690396    2543 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 07 02:18:06.690580 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.690503    2543 reconstruct.go:97] "Volume reconstruction finished"
Jul 07 02:18:06.690580 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.690512    2543 reconciler.go:29] "Reconciler: start to sync state"
Jul 07 02:18:06.691076 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.691058    2543 factory.go:55] Registering systemd factory
Jul 07 02:18:06.691143 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.691109    2543 factory.go:223] Registration of the systemd container factory successfully
Jul 07 02:18:06.691400 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.691387    2543 factory.go:153] Registering CRI-O factory
Jul 07 02:18:06.691400 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.691402    2543 factory.go:223] Registration of the crio container factory successfully
Jul 07 02:18:06.691526 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.691457    2543 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 07 02:18:06.691526 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.691483    2543 factory.go:103] Registering Raw factory
Jul 07 02:18:06.691526 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.691500    2543 manager.go:1196] Started watching for new ooms in manager
Jul 07 02:18:06.692164 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.692147    2543 manager.go:349] Starting recovery of all containers
Jul 07 02:18:06.695191 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.695163    2543 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 07 02:18:06.695321 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.695167    2543 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-101.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 07 02:18:06.698447 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.698423    2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c7f9l"
Jul 07 02:18:06.705497 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.705349    2543 manager.go:354] Recovery completed
Jul 07 02:18:06.709517 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.709505    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:06.711390 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.711372    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:06.711453 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.711405    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:06.711453 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.711414    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:06.711932 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.711919    2543 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 07 02:18:06.711932 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.711930    2543 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 07 02:18:06.712045 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.711945    2543 state_mem.go:36] "Initialized new in-memory state store"
Jul 07 02:18:06.712045 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.711987    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:06.712747 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.712734    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:06.712799 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.712758    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:06.712799 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.712771    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:06.713001 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.712989    2543 policy_none.go:49] "None policy: Start"
Jul 07 02:18:06.713053 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.713006    2543 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 07 02:18:06.713053 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.713017    2543 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 07 02:18:06.714063 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.714047    2543 policy_none.go:47] "Start"
Jul 07 02:18:06.763620 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.763594    2543 manager.go:337] "Starting Device Plugin manager"
Jul 07 02:18:06.763731 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.763635    2543 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 07 02:18:06.763731 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.763646    2543 server.go:87] "Starting device plugin registration server"
Jul 07 02:18:06.763981 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.763956    2543 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 07 02:18:06.764028 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.763987    2543 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 07 02:18:06.764108 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.764092    2543 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 07 02:18:06.764452 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.764209    2543 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 07 02:18:06.764452 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.764222    2543 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 07 02:18:06.764695 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.764673    2543 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 07 02:18:06.764781 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.764728    2543 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:06.864508 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.864434    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:06.865504 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.865486    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:06.865564 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.865516    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:06.865564 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.865527    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:06.865564 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.865552    2543 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-101.ec2.internal"
Jul 07 02:18:06.872150 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.872113    2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 07 02:18:06.872262 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.872197    2543 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-101.ec2.internal"
Jul 07 02:18:06.872262 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.872225    2543 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-101.ec2.internal\": node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:06.873462 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.873447    2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 07 02:18:06.873515 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.873482    2543 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 07 02:18:06.873515 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.873505    2543 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 07 02:18:06.873606 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.873538    2543 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 07 02:18:06.876540 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.875836    2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:18:06.929786 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:06.929763    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:06.973980 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.973939    2543 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal"]
Jul 07 02:18:06.974042 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.974023    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:06.975619 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.975605    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:06.975690 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.975636    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:06.975690 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.975650    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:06.976787 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.976775    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:06.976958 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.976943    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:06.977019 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.976993    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:06.977530 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.977513    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:06.977603 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.977547    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:06.977603 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.977560    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:06.977603 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.977566    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:06.977603 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.977585    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:06.977603 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.977594    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:06.979242 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.979228    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:06.979290 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.979252    2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 07 02:18:06.979887 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.979875    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientMemory"
Jul 07 02:18:06.980001 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.979899    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasNoDiskPressure"
Jul 07 02:18:06.980001 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.979909    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeHasSufficientPID"
Jul 07 02:18:06.992368 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.992342    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7423d652671212ee1b665b30f55640eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal\" (UID: \"7423d652671212ee1b665b30f55640eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:06.992468 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.992372    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7423d652671212ee1b665b30f55640eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal\" (UID: \"7423d652671212ee1b665b30f55640eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:06.992468 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:06.992390    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36ceb552a78de1cb2a021e04da828958-config\") pod \"kube-apiserver-proxy-ip-10-0-132-101.ec2.internal\" (UID: \"36ceb552a78de1cb2a021e04da828958\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.007668 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:07.007647    2543 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-101.ec2.internal\" not found" node="ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.012170 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:07.012152    2543 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-101.ec2.internal\" not found" node="ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.030584 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:07.030562    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:07.093042 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:07.093007    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7423d652671212ee1b665b30f55640eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal\" (UID: \"7423d652671212ee1b665b30f55640eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.093213 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:07.093049    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7423d652671212ee1b665b30f55640eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal\" (UID: \"7423d652671212ee1b665b30f55640eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.093213 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:07.093074    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36ceb552a78de1cb2a021e04da828958-config\") pod \"kube-apiserver-proxy-ip-10-0-132-101.ec2.internal\" (UID: \"36ceb552a78de1cb2a021e04da828958\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.093213 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:07.093122    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36ceb552a78de1cb2a021e04da828958-config\") pod \"kube-apiserver-proxy-ip-10-0-132-101.ec2.internal\" (UID: \"36ceb552a78de1cb2a021e04da828958\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.093213 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:07.093127    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7423d652671212ee1b665b30f55640eb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal\" (UID: \"7423d652671212ee1b665b30f55640eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.093213 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:07.093123    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7423d652671212ee1b665b30f55640eb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal\" (UID: \"7423d652671212ee1b665b30f55640eb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.131537 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:07.131460    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:07.232255 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:07.232211    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:07.310750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:07.310719    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.315728 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:07.315710    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:07.332505 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:07.332480    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:08.533289 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:08.533200    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:08.633778 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:08.633738    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:08.728238 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:08.728205    2543 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 07 02:18:08.728747 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:08.728390    2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 07 02:18:08.734457 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:08.734428    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:08.800614 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:08.800532    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-06 02:13:07 +0000 UTC" deadline="2028-04-10 22:28:16.22603817 +0000 UTC"
Jul 07 02:18:08.800614 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:08.800561    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15452h10m7.425480016s"
Jul 07 02:18:08.816617 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:08.816596    2543 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:18:08.834648 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:08.834625    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:08.935284 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:08.935253    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:08.987712 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:08.987662    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7423d652671212ee1b665b30f55640eb.slice/crio-775ed07fb5542c7e46fab113928dffd241cb84f8f9c2015c3d2034b6afe4d97e WatchSource:0}: Error finding container 775ed07fb5542c7e46fab113928dffd241cb84f8f9c2015c3d2034b6afe4d97e: Status 404 returned error can't find the container with id 775ed07fb5542c7e46fab113928dffd241cb84f8f9c2015c3d2034b6afe4d97e
Jul 07 02:18:08.988028 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:08.988008    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ceb552a78de1cb2a021e04da828958.slice/crio-59005a3aee7dd83628ea878c37d026736e08381de85f9898da0092b45f5cad9c WatchSource:0}: Error finding container 59005a3aee7dd83628ea878c37d026736e08381de85f9898da0092b45f5cad9c: Status 404 returned error can't find the container with id 59005a3aee7dd83628ea878c37d026736e08381de85f9898da0092b45f5cad9c
Jul 07 02:18:08.989025 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:08.988995    2543 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:18:08.993930 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:08.993917    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:18:09.035474 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.035448    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:09.136029 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.135960    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:09.236586 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.236556    2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-101.ec2.internal\" not found"
Jul 07 02:18:09.256179 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.256157    2543 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:18:09.290446 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.290420    2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:09.304232 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.304216    2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 02:18:09.306626 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.306613    2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal"
Jul 07 02:18:09.317267 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.317246    2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 07 02:18:09.770362 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.770330    2543 apiserver.go:52] "Watching apiserver"
Jul 07 02:18:09.777159 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.777135    2543 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 07 02:18:09.777579 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.777554    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-6tqbv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal","openshift-multus/network-metrics-daemon-wllmf","openshift-network-diagnostics/network-check-target-6kx2t","openshift-network-operator/iptables-alerter-tqlvt","openshift-ovn-kubernetes/ovnkube-node-gh2mp","kube-system/global-pull-secret-syncer-drrrz","kube-system/konnectivity-agent-vkx8c","kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm","openshift-dns/node-resolver-nqvp2","openshift-image-registry/node-ca-6l9zx","openshift-multus/multus-additional-cni-plugins-sdd9x","openshift-multus/multus-m7vbn"]
Jul 07 02:18:09.779261 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.779239    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.781089 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.781067    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:09.781668 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.781651    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 07 02:18:09.781886 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.781871    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 07 02:18:09.781998 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.781979    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 07 02:18:09.782122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.782109    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 07 02:18:09.782217 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.782123    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-284ss\""
Jul 07 02:18:09.782280 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.782226    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 07 02:18:09.782473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.782455    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 07 02:18:09.783230 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.783213    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:09.783430 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.783409    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:09.784907 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.784886    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 07 02:18:09.785404 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.785386    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w25kw\""
Jul 07 02:18:09.786142 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.785638    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 07 02:18:09.786142 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.785869    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:09.786142 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.785936    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:09.787618 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.787599    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:09.789452 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.789428    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:09.789546 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.789511    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:09.790256 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.790238    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pscfz\""
Jul 07 02:18:09.790433 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.790418    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:18:09.790619 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.790600    2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 07 02:18:09.790683 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.790664    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 07 02:18:09.790758 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.790740    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 07 02:18:09.791303 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.791286    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.792905 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.792674    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:09.793593 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.793578    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:18:09.793671 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.793581    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-j5zdj\""
Jul 07 02:18:09.793851 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.793837    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 07 02:18:09.794148 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.794133    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:09.794814 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.794797    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 07 02:18:09.795454 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.795433    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 07 02:18:09.795548 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.795468    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m2c4p\""
Jul 07 02:18:09.795548 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.795434    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 07 02:18:09.796540 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.796521    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 07 02:18:09.796625 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.796584    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r6ptz\""
Jul 07 02:18:09.796877 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.796861    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:09.797984 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.797968    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 07 02:18:09.799110 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.799094    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 07 02:18:09.799617 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.799601    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 07 02:18:09.799885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.799871    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mqm9k\""
Jul 07 02:18:09.800113 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.800049    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 07 02:18:09.800268 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.800255    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:09.801078 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801047    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.801793 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801773    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-sys\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.801877 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801811    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-run-netns\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.801877 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801835    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-run-ovn\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.801877 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801856    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-log-socket\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801877    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de69c094-35c2-4b5a-8d70-efdc95adec5b-env-overrides\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801903    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/352d6441-d9b5-49c7-a3b3-e249addf652b-host-slash\") pod \"iptables-alerter-tqlvt\" (UID: \"352d6441-d9b5-49c7-a3b3-e249addf652b\") " pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:09.802038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801927    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8pqb\" (UniqueName: \"kubernetes.io/projected/352d6441-d9b5-49c7-a3b3-e249addf652b-kube-api-access-d8pqb\") pod \"iptables-alerter-tqlvt\" (UID: \"352d6441-d9b5-49c7-a3b3-e249addf652b\") " pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:09.802038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801951    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-sysconfig\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801973    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-slash\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.801996    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6fb4b23e-61ae-449e-abad-c58f2c9f5bc6-konnectivity-ca\") pod \"konnectivity-agent-vkx8c\" (UID: \"6fb4b23e-61ae-449e-abad-c58f2c9f5bc6\") " pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:09.802038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802017    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-lib-modules\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802039    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-host\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802061    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-kubelet\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802084    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-node-log\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802114    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802145    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de69c094-35c2-4b5a-8d70-efdc95adec5b-ovnkube-config\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802167    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p764r\" (UniqueName: \"kubernetes.io/projected/de69c094-35c2-4b5a-8d70-efdc95adec5b-kube-api-access-p764r\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802218    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-sysctl-d\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802242    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2tp5\" (UniqueName: \"kubernetes.io/projected/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-kube-api-access-w2tp5\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802266    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-systemd-units\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802294    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-etc-openvswitch\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802309    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-cni-netd\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802324    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6fb4b23e-61ae-449e-abad-c58f2c9f5bc6-agent-certs\") pod \"konnectivity-agent-vkx8c\" (UID: \"6fb4b23e-61ae-449e-abad-c58f2c9f5bc6\") " pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:09.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802339    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-sysctl-conf\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802397    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-tuned\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802431    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-tmp\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802455    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802479    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/352d6441-d9b5-49c7-a3b3-e249addf652b-iptables-alerter-script\") pod \"iptables-alerter-tqlvt\" (UID: \"352d6441-d9b5-49c7-a3b3-e249addf652b\") " pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802502    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-kubernetes\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802513    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802547    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-run\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802598    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-var-lib-kubelet\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802630    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-var-lib-openvswitch\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802646    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802662    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-cni-bin\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802687    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-systemd\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802710    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfd04d15-d522-4e36-83ff-1631d6f859aa-dbus\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802743    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-run-openvswitch\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802767    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802798    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802806    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-modprobe-d\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802821    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 07 02:18:09.802925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802826    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vzl5g\""
Jul 07 02:18:09.803754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802835    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfd04d15-d522-4e36-83ff-1631d6f859aa-kubelet-config\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:09.803754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802862    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:09.803754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802884    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-run-systemd\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.803754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802909    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 07 02:18:09.803754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802922    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de69c094-35c2-4b5a-8d70-efdc95adec5b-ovn-node-metrics-cert\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.803754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802954    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de69c094-35c2-4b5a-8d70-efdc95adec5b-ovnkube-script-lib\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.803754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802979    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzg9q\" (UniqueName: \"kubernetes.io/projected/50534db7-9f97-4088-bc8a-b33de25d0d36-kube-api-access-gzg9q\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:09.803754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.802997    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqdjb\" (UniqueName: \"kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb\") pod \"network-check-target-6kx2t\" (UID: \"47733a2c-fd25-4bfe-9883-84726a7029d8\") " pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:09.803754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.803217    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 07 02:18:09.803754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.803347    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zldg6\""
Jul 07 02:18:09.804656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.804636    2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 07 02:18:09.829743 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.829713    2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wqpff"
Jul 07 02:18:09.843447 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.843376    2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wqpff"
Jul 07 02:18:09.891341 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.891314    2543 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904121    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p764r\" (UniqueName: \"kubernetes.io/projected/de69c094-35c2-4b5a-8d70-efdc95adec5b-kube-api-access-p764r\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904170    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-sysctl-d\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904212    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-cni-netd\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904236    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6fb4b23e-61ae-449e-abad-c58f2c9f5bc6-agent-certs\") pod \"konnectivity-agent-vkx8c\" (UID: \"6fb4b23e-61ae-449e-abad-c58f2c9f5bc6\") " pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904258    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-tmp\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904280    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904301    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/352d6441-d9b5-49c7-a3b3-e249addf652b-iptables-alerter-script\") pod \"iptables-alerter-tqlvt\" (UID: \"352d6441-d9b5-49c7-a3b3-e249addf652b\") " pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904325    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-cni-bin\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904351    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfd04d15-d522-4e36-83ff-1631d6f859aa-dbus\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904378    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-cni-binary-copy\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904402    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-var-lib-cni-bin\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904427    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gskc5\" (UniqueName: \"kubernetes.io/projected/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-kube-api-access-gskc5\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.904428    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904451    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfd04d15-d522-4e36-83ff-1631d6f859aa-kubelet-config\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904473    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-run-netns\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904489    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-cni-bin\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.905017 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.904503    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs podName:50534db7-9f97-4088-bc8a-b33de25d0d36 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:10.404481079 +0000 UTC m=+3.047641615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs") pod "network-metrics-daemon-wllmf" (UID: "50534db7-9f97-4088-bc8a-b33de25d0d36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904554    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-cni-netd\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904562    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-var-lib-kubelet\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904602    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdjb\" (UniqueName: \"kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb\") pod \"network-check-target-6kx2t\" (UID: \"47733a2c-fd25-4bfe-9883-84726a7029d8\") " pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904619    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfd04d15-d522-4e36-83ff-1631d6f859aa-kubelet-config\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904619    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfd04d15-d522-4e36-83ff-1631d6f859aa-dbus\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904351    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-sysctl-d\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904634    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl7x7\" (UniqueName: \"kubernetes.io/projected/8840d1c9-665e-45fb-b8f3-1bf632053a01-kube-api-access-rl7x7\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904674    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-os-release\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904717    2543 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904779    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-run-ovn\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904816    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-log-socket\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904858    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-log-socket\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904867    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-run-ovn\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904890    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8pqb\" (UniqueName: \"kubernetes.io/projected/352d6441-d9b5-49c7-a3b3-e249addf652b-kube-api-access-d8pqb\") pod \"iptables-alerter-tqlvt\" (UID: \"352d6441-d9b5-49c7-a3b3-e249addf652b\") " pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904901    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/352d6441-d9b5-49c7-a3b3-e249addf652b-iptables-alerter-script\") pod \"iptables-alerter-tqlvt\" (UID: \"352d6441-d9b5-49c7-a3b3-e249addf652b\") " pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904926    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-sysconfig\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.905846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904964    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c703c034-4b57-4d22-a865-01ab46b75790-serviceca\") pod \"node-ca-6l9zx\" (UID: \"c703c034-4b57-4d22-a865-01ab46b75790\") " pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904978    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-sysconfig\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.904990    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905023    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-slash\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905044    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-host\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905072    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-daemon-config\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905097    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-run-multus-certs\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905105    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-host\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905099    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-slash\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905120    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e052eaa2-eca7-4894-8be6-e6b273851fa7-hosts-file\") pod \"node-resolver-nqvp2\" (UID: \"e052eaa2-eca7-4894-8be6-e6b273851fa7\") " pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905159    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de69c094-35c2-4b5a-8d70-efdc95adec5b-ovnkube-config\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905205    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2tp5\" (UniqueName: \"kubernetes.io/projected/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-kube-api-access-w2tp5\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905234    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-var-lib-cni-multus\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905256    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-systemd-units\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905277    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-etc-openvswitch\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905291    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-sysctl-conf\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905307    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-tuned\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.906513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905329    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-hostroot\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905352    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-socket-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905376    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-kubernetes\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905398    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-run\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905424    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-var-lib-kubelet\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905448    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l6zn\" (UniqueName: \"kubernetes.io/projected/e052eaa2-eca7-4894-8be6-e6b273851fa7-kube-api-access-8l6zn\") pod \"node-resolver-nqvp2\" (UID: \"e052eaa2-eca7-4894-8be6-e6b273851fa7\") " pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905472    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-cnibin\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905499    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bca64acd-88bf-4c2e-9677-50add32ca830-cni-binary-copy\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905522    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-var-lib-openvswitch\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905545    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-systemd\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905570    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-system-cni-dir\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905592    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-cni-dir\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905613    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-os-release\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905638    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-socket-dir-parent\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905662    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-sys-fs\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905679    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de69c094-35c2-4b5a-8d70-efdc95adec5b-ovnkube-config\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905684    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c703c034-4b57-4d22-a865-01ab46b75790-host\") pod \"node-ca-6l9zx\" (UID: \"c703c034-4b57-4d22-a865-01ab46b75790\") " pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:09.907316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905814    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-etc-openvswitch\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905891    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-systemd-units\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905896    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-sysctl-conf\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905942    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-kubernetes\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905971    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-run\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.905996    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-var-lib-kubelet\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906031    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-var-lib-openvswitch\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906072    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-systemd\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906141    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-run-openvswitch\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906179    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906225    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-modprobe-d\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906262    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906288    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-conf-dir\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906313    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-system-cni-dir\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906340    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bca64acd-88bf-4c2e-9677-50add32ca830-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906373    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqfzb\" (UniqueName: \"kubernetes.io/projected/bca64acd-88bf-4c2e-9677-50add32ca830-kube-api-access-xqfzb\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906389    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-run-systemd\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.908122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906410    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de69c094-35c2-4b5a-8d70-efdc95adec5b-ovn-node-metrics-cert\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906428    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de69c094-35c2-4b5a-8d70-efdc95adec5b-ovnkube-script-lib\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906443    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzg9q\" (UniqueName: \"kubernetes.io/projected/50534db7-9f97-4088-bc8a-b33de25d0d36-kube-api-access-gzg9q\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906458    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-sys\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906473    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906489    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h26mw\" (UniqueName: \"kubernetes.io/projected/c703c034-4b57-4d22-a865-01ab46b75790-kube-api-access-h26mw\") pod \"node-ca-6l9zx\" (UID: \"c703c034-4b57-4d22-a865-01ab46b75790\") " pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906527    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bca64acd-88bf-4c2e-9677-50add32ca830-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906546    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-run-netns\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906560    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de69c094-35c2-4b5a-8d70-efdc95adec5b-env-overrides\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906576    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/352d6441-d9b5-49c7-a3b3-e249addf652b-host-slash\") pod \"iptables-alerter-tqlvt\" (UID: \"352d6441-d9b5-49c7-a3b3-e249addf652b\") " pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906593    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-run-k8s-cni-cncf-io\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906607    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-etc-kubernetes\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906621    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906636    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-registration-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906650    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6fb4b23e-61ae-449e-abad-c58f2c9f5bc6-konnectivity-ca\") pod \"konnectivity-agent-vkx8c\" (UID: \"6fb4b23e-61ae-449e-abad-c58f2c9f5bc6\") " pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906666    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-lib-modules\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906681    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-cnibin\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:09.908885 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906696    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-device-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906712    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e052eaa2-eca7-4894-8be6-e6b273851fa7-tmp-dir\") pod \"node-resolver-nqvp2\" (UID: \"e052eaa2-eca7-4894-8be6-e6b273851fa7\") " pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906727    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-kubelet\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906742    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-node-log\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906819    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906867    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906887    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-node-log\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906943    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-modprobe-d\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906966    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-run-openvswitch\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.906992    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-run-ovn-kubernetes\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.907042    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.907083    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret podName:dfd04d15-d522-4e36-83ff-1631d6f859aa nodeName:}" failed. No retries permitted until 2026-07-07 02:18:10.407072312 +0000 UTC m=+3.050232812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret") pod "global-pull-secret-syncer-drrrz" (UID: "dfd04d15-d522-4e36-83ff-1631d6f859aa") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.907109    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-run-systemd\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.907795    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de69c094-35c2-4b5a-8d70-efdc95adec5b-ovnkube-script-lib\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.908054    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-sys\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.908099    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-run-netns\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.908608    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de69c094-35c2-4b5a-8d70-efdc95adec5b-env-overrides\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.909739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.908659    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/352d6441-d9b5-49c7-a3b3-e249addf652b-host-slash\") pod \"iptables-alerter-tqlvt\" (UID: \"352d6441-d9b5-49c7-a3b3-e249addf652b\") " pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:09.910500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.908756    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-lib-modules\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.910500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.908844    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-etc-tuned\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.910500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.909268    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6fb4b23e-61ae-449e-abad-c58f2c9f5bc6-konnectivity-ca\") pod \"konnectivity-agent-vkx8c\" (UID: \"6fb4b23e-61ae-449e-abad-c58f2c9f5bc6\") " pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:09.910500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.909353    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de69c094-35c2-4b5a-8d70-efdc95adec5b-host-kubelet\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.910500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.910258    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de69c094-35c2-4b5a-8d70-efdc95adec5b-ovn-node-metrics-cert\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.911387 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.911365    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6fb4b23e-61ae-449e-abad-c58f2c9f5bc6-agent-certs\") pod \"konnectivity-agent-vkx8c\" (UID: \"6fb4b23e-61ae-449e-abad-c58f2c9f5bc6\") " pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:09.912835 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.912813    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-tmp\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.913057 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.913040    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:09.913171 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.913063    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:09.913171 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.913078    2543 projected.go:196] Error preparing data for projected volume kube-api-access-cqdjb for pod openshift-network-diagnostics/network-check-target-6kx2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:09.913171 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:09.913136    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb podName:47733a2c-fd25-4bfe-9883-84726a7029d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:10.413120346 +0000 UTC m=+3.056280867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cqdjb" (UniqueName: "kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb") pod "network-check-target-6kx2t" (UID: "47733a2c-fd25-4bfe-9883-84726a7029d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:09.915478 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.915454    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p764r\" (UniqueName: \"kubernetes.io/projected/de69c094-35c2-4b5a-8d70-efdc95adec5b-kube-api-access-p764r\") pod \"ovnkube-node-gh2mp\" (UID: \"de69c094-35c2-4b5a-8d70-efdc95adec5b\") " pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:09.915975 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.915907    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8pqb\" (UniqueName: \"kubernetes.io/projected/352d6441-d9b5-49c7-a3b3-e249addf652b-kube-api-access-d8pqb\") pod \"iptables-alerter-tqlvt\" (UID: \"352d6441-d9b5-49c7-a3b3-e249addf652b\") " pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:09.916608 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.916564    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2tp5\" (UniqueName: \"kubernetes.io/projected/910a95ac-f4c9-42e9-ae74-c22ed25c6be0-kube-api-access-w2tp5\") pod \"tuned-6tqbv\" (UID: \"910a95ac-f4c9-42e9-ae74-c22ed25c6be0\") " pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:09.917131 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.917110    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzg9q\" (UniqueName: \"kubernetes.io/projected/50534db7-9f97-4088-bc8a-b33de25d0d36-kube-api-access-gzg9q\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:09.979060 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.979008    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal" event={"ID":"7423d652671212ee1b665b30f55640eb","Type":"ContainerStarted","Data":"775ed07fb5542c7e46fab113928dffd241cb84f8f9c2015c3d2034b6afe4d97e"}
Jul 07 02:18:09.980305 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:09.980273    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal" event={"ID":"36ceb552a78de1cb2a021e04da828958","Type":"ContainerStarted","Data":"59005a3aee7dd83628ea878c37d026736e08381de85f9898da0092b45f5cad9c"}
Jul 07 02:18:10.008148 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008120    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c703c034-4b57-4d22-a865-01ab46b75790-serviceca\") pod \"node-ca-6l9zx\" (UID: \"c703c034-4b57-4d22-a865-01ab46b75790\") " pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:10.008287 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008164    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.008287 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008211    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-daemon-config\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008287 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008233    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-run-multus-certs\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008287 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008254    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e052eaa2-eca7-4894-8be6-e6b273851fa7-hosts-file\") pod \"node-resolver-nqvp2\" (UID: \"e052eaa2-eca7-4894-8be6-e6b273851fa7\") " pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:10.008287 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008276    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-var-lib-cni-multus\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008301    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-hostroot\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008323    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-socket-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.008541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008346    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l6zn\" (UniqueName: \"kubernetes.io/projected/e052eaa2-eca7-4894-8be6-e6b273851fa7-kube-api-access-8l6zn\") pod \"node-resolver-nqvp2\" (UID: \"e052eaa2-eca7-4894-8be6-e6b273851fa7\") " pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:10.008541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008395    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-cnibin\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.008541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008401    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.008541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008423    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bca64acd-88bf-4c2e-9677-50add32ca830-cni-binary-copy\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.008541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008465    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-system-cni-dir\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008492    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-cni-dir\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008514    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-os-release\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008540    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-socket-dir-parent\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008550    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-hostroot\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008567    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-sys-fs\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008590    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c703c034-4b57-4d22-a865-01ab46b75790-host\") pod \"node-ca-6l9zx\" (UID: \"c703c034-4b57-4d22-a865-01ab46b75790\") " pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008598    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-run-multus-certs\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008637    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c703c034-4b57-4d22-a865-01ab46b75790-host\") pod \"node-ca-6l9zx\" (UID: \"c703c034-4b57-4d22-a865-01ab46b75790\") " pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008639    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-conf-dir\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008666    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-system-cni-dir\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008692    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bca64acd-88bf-4c2e-9677-50add32ca830-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008714    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqfzb\" (UniqueName: \"kubernetes.io/projected/bca64acd-88bf-4c2e-9677-50add32ca830-kube-api-access-xqfzb\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008741    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008756    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c703c034-4b57-4d22-a865-01ab46b75790-serviceca\") pod \"node-ca-6l9zx\" (UID: \"c703c034-4b57-4d22-a865-01ab46b75790\") " pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008765    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h26mw\" (UniqueName: \"kubernetes.io/projected/c703c034-4b57-4d22-a865-01ab46b75790-kube-api-access-h26mw\") pod \"node-ca-6l9zx\" (UID: \"c703c034-4b57-4d22-a865-01ab46b75790\") " pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008790    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bca64acd-88bf-4c2e-9677-50add32ca830-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008814    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-var-lib-cni-multus\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008858    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-run-k8s-cni-cncf-io\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008865    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-system-cni-dir\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.008920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008819    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-run-k8s-cni-cncf-io\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008882    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-socket-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008898    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-etc-kubernetes\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008668    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-conf-dir\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008894    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-daemon-config\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008923    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-cnibin\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008944    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008955    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-etc-kubernetes\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008967    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-socket-dir-parent\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008974    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-registration-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009011    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-multus-cni-dir\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009024    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-os-release\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009024    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bca64acd-88bf-4c2e-9677-50add32ca830-cni-binary-copy\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009057    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-cnibin\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009063    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009005    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-cnibin\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.008766    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e052eaa2-eca7-4894-8be6-e6b273851fa7-hosts-file\") pod \"node-resolver-nqvp2\" (UID: \"e052eaa2-eca7-4894-8be6-e6b273851fa7\") " pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009087    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-registration-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.009697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009096    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-device-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009126    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-etc-selinux\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009156    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-sys-fs\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009161    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8840d1c9-665e-45fb-b8f3-1bf632053a01-device-dir\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009227    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-system-cni-dir\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009256    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e052eaa2-eca7-4894-8be6-e6b273851fa7-tmp-dir\") pod \"node-resolver-nqvp2\" (UID: \"e052eaa2-eca7-4894-8be6-e6b273851fa7\") " pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009306    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-cni-binary-copy\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009335    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-var-lib-cni-bin\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009359    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gskc5\" (UniqueName: \"kubernetes.io/projected/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-kube-api-access-gskc5\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009388    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-run-netns\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009413    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-var-lib-kubelet\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009415    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-var-lib-cni-bin\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009457    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-var-lib-kubelet\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009469    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-host-run-netns\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009565    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bca64acd-88bf-4c2e-9677-50add32ca830-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009591    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e052eaa2-eca7-4894-8be6-e6b273851fa7-tmp-dir\") pod \"node-resolver-nqvp2\" (UID: \"e052eaa2-eca7-4894-8be6-e6b273851fa7\") " pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009608    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bca64acd-88bf-4c2e-9677-50add32ca830-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.010502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009619    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl7x7\" (UniqueName: \"kubernetes.io/projected/8840d1c9-665e-45fb-b8f3-1bf632053a01-kube-api-access-rl7x7\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.011132 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009656    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-os-release\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.011132 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009719    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bca64acd-88bf-4c2e-9677-50add32ca830-os-release\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.011132 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.009925    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-cni-binary-copy\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.019436 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.019415    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l6zn\" (UniqueName: \"kubernetes.io/projected/e052eaa2-eca7-4894-8be6-e6b273851fa7-kube-api-access-8l6zn\") pod \"node-resolver-nqvp2\" (UID: \"e052eaa2-eca7-4894-8be6-e6b273851fa7\") " pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:10.019546 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.019446    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h26mw\" (UniqueName: \"kubernetes.io/projected/c703c034-4b57-4d22-a865-01ab46b75790-kube-api-access-h26mw\") pod \"node-ca-6l9zx\" (UID: \"c703c034-4b57-4d22-a865-01ab46b75790\") " pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:10.021200 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.021123    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl7x7\" (UniqueName: \"kubernetes.io/projected/8840d1c9-665e-45fb-b8f3-1bf632053a01-kube-api-access-rl7x7\") pod \"aws-ebs-csi-driver-node-2cmtm\" (UID: \"8840d1c9-665e-45fb-b8f3-1bf632053a01\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.021576 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.021555    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gskc5\" (UniqueName: \"kubernetes.io/projected/5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e-kube-api-access-gskc5\") pod \"multus-m7vbn\" (UID: \"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e\") " pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.021655 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.021583    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqfzb\" (UniqueName: \"kubernetes.io/projected/bca64acd-88bf-4c2e-9677-50add32ca830-kube-api-access-xqfzb\") pod \"multus-additional-cni-plugins-sdd9x\" (UID: \"bca64acd-88bf-4c2e-9677-50add32ca830\") " pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.092336 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.092304    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:10.099464 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.099436    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:10.109274 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.109251    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tqlvt"
Jul 07 02:18:10.115277 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.115255    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6tqbv"
Jul 07 02:18:10.117391 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.117373    2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 07 02:18:10.123584 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.123566    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm"
Jul 07 02:18:10.129257 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.129241    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nqvp2"
Jul 07 02:18:10.135940 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.135924    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6l9zx"
Jul 07 02:18:10.148306 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.148282    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sdd9x"
Jul 07 02:18:10.154638 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.154613    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m7vbn"
Jul 07 02:18:10.412778 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.412701    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:10.412778 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.412754    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:10.412966 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:10.412889    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:10.412966 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:10.412955    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret podName:dfd04d15-d522-4e36-83ff-1631d6f859aa nodeName:}" failed. No retries permitted until 2026-07-07 02:18:11.412940619 +0000 UTC m=+4.056101119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret") pod "global-pull-secret-syncer-drrrz" (UID: "dfd04d15-d522-4e36-83ff-1631d6f859aa") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:10.413044 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:10.412889    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:10.413044 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:10.413034    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs podName:50534db7-9f97-4088-bc8a-b33de25d0d36 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:11.413021766 +0000 UTC m=+4.056182265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs") pod "network-metrics-daemon-wllmf" (UID: "50534db7-9f97-4088-bc8a-b33de25d0d36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:10.513894 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.513861    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdjb\" (UniqueName: \"kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb\") pod \"network-check-target-6kx2t\" (UID: \"47733a2c-fd25-4bfe-9883-84726a7029d8\") " pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:10.514044 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:10.514024    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:10.514044 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:10.514044    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:10.514122 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:10.514054    2543 projected.go:196] Error preparing data for projected volume kube-api-access-cqdjb for pod openshift-network-diagnostics/network-check-target-6kx2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:10.514122 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:10.514107    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb podName:47733a2c-fd25-4bfe-9883-84726a7029d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:11.514090641 +0000 UTC m=+4.157251143 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqdjb" (UniqueName: "kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb") pod "network-check-target-6kx2t" (UID: "47733a2c-fd25-4bfe-9883-84726a7029d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:10.846657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.846599    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:13:09 +0000 UTC" deadline="2028-02-20 12:52:14.414522003 +0000 UTC"
Jul 07 02:18:10.846657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.846636    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14242h34m3.567890968s"
Jul 07 02:18:10.908827 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:10.908801    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca64acd_88bf_4c2e_9677_50add32ca830.slice/crio-f2f2f6a02a615f0c83de5b9fdc434e8b51e9256ea2865202ef1804bab92e5c72 WatchSource:0}: Error finding container f2f2f6a02a615f0c83de5b9fdc434e8b51e9256ea2865202ef1804bab92e5c72: Status 404 returned error can't find the container with id f2f2f6a02a615f0c83de5b9fdc434e8b51e9256ea2865202ef1804bab92e5c72
Jul 07 02:18:10.910770 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:10.910745    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910a95ac_f4c9_42e9_ae74_c22ed25c6be0.slice/crio-4ece1668268c605496128a104c6edc34d9cc001df11ccf441a42b6231e25e5e1 WatchSource:0}: Error finding container 4ece1668268c605496128a104c6edc34d9cc001df11ccf441a42b6231e25e5e1: Status 404 returned error can't find the container with id 4ece1668268c605496128a104c6edc34d9cc001df11ccf441a42b6231e25e5e1
Jul 07 02:18:10.914071 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:10.914050    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352d6441_d9b5_49c7_a3b3_e249addf652b.slice/crio-6a46a088b7efae0d8115fe5a95344a24477bba4a6ba8cf3e67e706322a0215f6 WatchSource:0}: Error finding container 6a46a088b7efae0d8115fe5a95344a24477bba4a6ba8cf3e67e706322a0215f6: Status 404 returned error can't find the container with id 6a46a088b7efae0d8115fe5a95344a24477bba4a6ba8cf3e67e706322a0215f6
Jul 07 02:18:10.915249 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:10.915154    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc703c034_4b57_4d22_a865_01ab46b75790.slice/crio-a9992da53da207b07b8630f3db3032d59d0a3ce7deda03a003062d833ae66784 WatchSource:0}: Error finding container a9992da53da207b07b8630f3db3032d59d0a3ce7deda03a003062d833ae66784: Status 404 returned error can't find the container with id a9992da53da207b07b8630f3db3032d59d0a3ce7deda03a003062d833ae66784
Jul 07 02:18:10.916052 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:10.916034    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde69c094_35c2_4b5a_8d70_efdc95adec5b.slice/crio-b8edd3c4cf5b8bad1835bad4bb20af6ddc5410f59e44d81d0e0ff6bb9ec44cff WatchSource:0}: Error finding container b8edd3c4cf5b8bad1835bad4bb20af6ddc5410f59e44d81d0e0ff6bb9ec44cff: Status 404 returned error can't find the container with id b8edd3c4cf5b8bad1835bad4bb20af6ddc5410f59e44d81d0e0ff6bb9ec44cff
Jul 07 02:18:10.917081 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:10.917043    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf5689f_ebb4_4e70_bce5_6680ea8dcd3e.slice/crio-9820c0737003e47a232aaca09f5f708ecaf59f9cc41fd7251cf1df6c7d2986d8 WatchSource:0}: Error finding container 9820c0737003e47a232aaca09f5f708ecaf59f9cc41fd7251cf1df6c7d2986d8: Status 404 returned error can't find the container with id 9820c0737003e47a232aaca09f5f708ecaf59f9cc41fd7251cf1df6c7d2986d8
Jul 07 02:18:10.919437 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:10.919392    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode052eaa2_eca7_4894_8be6_e6b273851fa7.slice/crio-166f6711564aa5749cf1900b55c7f9d013b75ee18f140351aff57f0fe9518cf1 WatchSource:0}: Error finding container 166f6711564aa5749cf1900b55c7f9d013b75ee18f140351aff57f0fe9518cf1: Status 404 returned error can't find the container with id 166f6711564aa5749cf1900b55c7f9d013b75ee18f140351aff57f0fe9518cf1
Jul 07 02:18:10.920652 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:10.920619    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8840d1c9_665e_45fb_b8f3_1bf632053a01.slice/crio-44de92fe7c8849e71052a4060bad45d292f20869e3a73a0b488251bfdd4553f8 WatchSource:0}: Error finding container 44de92fe7c8849e71052a4060bad45d292f20869e3a73a0b488251bfdd4553f8: Status 404 returned error can't find the container with id 44de92fe7c8849e71052a4060bad45d292f20869e3a73a0b488251bfdd4553f8
Jul 07 02:18:10.926356 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:10.926339    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb4b23e_61ae_449e_abad_c58f2c9f5bc6.slice/crio-64de0f7f05ae5be235b1ed07268b9d809e6919c48c83c6e8a9e02fa304078370 WatchSource:0}: Error finding container 64de0f7f05ae5be235b1ed07268b9d809e6919c48c83c6e8a9e02fa304078370: Status 404 returned error can't find the container with id 64de0f7f05ae5be235b1ed07268b9d809e6919c48c83c6e8a9e02fa304078370
Jul 07 02:18:10.974802 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.974651    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:10.974933 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.974687    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:10.974933 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:10.974910    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:10.975041 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:10.974977    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:10.982751 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.982728    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkx8c" event={"ID":"6fb4b23e-61ae-449e-abad-c58f2c9f5bc6","Type":"ContainerStarted","Data":"64de0f7f05ae5be235b1ed07268b9d809e6919c48c83c6e8a9e02fa304078370"}
Jul 07 02:18:10.983931 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.983909    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm" event={"ID":"8840d1c9-665e-45fb-b8f3-1bf632053a01","Type":"ContainerStarted","Data":"44de92fe7c8849e71052a4060bad45d292f20869e3a73a0b488251bfdd4553f8"}
Jul 07 02:18:10.984948 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.984928    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m7vbn" event={"ID":"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e","Type":"ContainerStarted","Data":"9820c0737003e47a232aaca09f5f708ecaf59f9cc41fd7251cf1df6c7d2986d8"}
Jul 07 02:18:10.986026 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.986007    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp" event={"ID":"de69c094-35c2-4b5a-8d70-efdc95adec5b","Type":"ContainerStarted","Data":"b8edd3c4cf5b8bad1835bad4bb20af6ddc5410f59e44d81d0e0ff6bb9ec44cff"}
Jul 07 02:18:10.987286 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.987264    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nqvp2" event={"ID":"e052eaa2-eca7-4894-8be6-e6b273851fa7","Type":"ContainerStarted","Data":"166f6711564aa5749cf1900b55c7f9d013b75ee18f140351aff57f0fe9518cf1"}
Jul 07 02:18:10.988537 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.988511    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6l9zx" event={"ID":"c703c034-4b57-4d22-a865-01ab46b75790","Type":"ContainerStarted","Data":"a9992da53da207b07b8630f3db3032d59d0a3ce7deda03a003062d833ae66784"}
Jul 07 02:18:10.989618 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.989595    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tqlvt" event={"ID":"352d6441-d9b5-49c7-a3b3-e249addf652b","Type":"ContainerStarted","Data":"6a46a088b7efae0d8115fe5a95344a24477bba4a6ba8cf3e67e706322a0215f6"}
Jul 07 02:18:10.990786 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.990752    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6tqbv" event={"ID":"910a95ac-f4c9-42e9-ae74-c22ed25c6be0","Type":"ContainerStarted","Data":"4ece1668268c605496128a104c6edc34d9cc001df11ccf441a42b6231e25e5e1"}
Jul 07 02:18:10.991767 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:10.991743    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdd9x" event={"ID":"bca64acd-88bf-4c2e-9677-50add32ca830","Type":"ContainerStarted","Data":"f2f2f6a02a615f0c83de5b9fdc434e8b51e9256ea2865202ef1804bab92e5c72"}
Jul 07 02:18:11.419735 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:11.419682    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:11.419918 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:11.419779    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:11.419981 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:11.419944    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:11.420036 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:11.420009    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret podName:dfd04d15-d522-4e36-83ff-1631d6f859aa nodeName:}" failed. No retries permitted until 2026-07-07 02:18:13.419990596 +0000 UTC m=+6.063151112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret") pod "global-pull-secret-syncer-drrrz" (UID: "dfd04d15-d522-4e36-83ff-1631d6f859aa") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:11.420483 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:11.420462    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:11.420579 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:11.420519    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs podName:50534db7-9f97-4088-bc8a-b33de25d0d36 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:13.420503349 +0000 UTC m=+6.063663864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs") pod "network-metrics-daemon-wllmf" (UID: "50534db7-9f97-4088-bc8a-b33de25d0d36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:11.520863 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:11.520777    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdjb\" (UniqueName: \"kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb\") pod \"network-check-target-6kx2t\" (UID: \"47733a2c-fd25-4bfe-9883-84726a7029d8\") " pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:11.521027 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:11.520957    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:11.521027 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:11.520983    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:11.521027 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:11.520995    2543 projected.go:196] Error preparing data for projected volume kube-api-access-cqdjb for pod openshift-network-diagnostics/network-check-target-6kx2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:11.521213 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:11.521054    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb podName:47733a2c-fd25-4bfe-9883-84726a7029d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:13.521036223 +0000 UTC m=+6.164196737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqdjb" (UniqueName: "kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb") pod "network-check-target-6kx2t" (UID: "47733a2c-fd25-4bfe-9883-84726a7029d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:11.847354 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:11.847236    2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-06 02:13:09 +0000 UTC" deadline="2028-02-20 08:34:08.049985402 +0000 UTC"
Jul 07 02:18:11.847354 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:11.847298    2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14238h15m56.202691811s"
Jul 07 02:18:11.977474 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:11.976981    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:11.977474 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:11.977098    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:12.004851 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:12.004814    2543 generic.go:356] "Generic (PLEG): container finished" podID="7423d652671212ee1b665b30f55640eb" containerID="ac3f28bd6febe21430d83d5be41cd5a002084b9f8b2569318c2f8985fe6c406d" exitCode=0
Jul 07 02:18:12.005009 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:12.004898    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal" event={"ID":"7423d652671212ee1b665b30f55640eb","Type":"ContainerDied","Data":"ac3f28bd6febe21430d83d5be41cd5a002084b9f8b2569318c2f8985fe6c406d"}
Jul 07 02:18:12.020820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:12.020393    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal" event={"ID":"36ceb552a78de1cb2a021e04da828958","Type":"ContainerStarted","Data":"99ea10db3f8ea6c4cfc6d70c9149a6fad00b4ae93865b8397488ae043d6fd1cf"}
Jul 07 02:18:12.974343 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:12.974311    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:12.974787 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:12.974446    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:12.974844 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:12.974826    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:12.974942 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:12.974918    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:13.035839 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:13.035804    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal" event={"ID":"7423d652671212ee1b665b30f55640eb","Type":"ContainerStarted","Data":"03ddbad2926114008808a17478dda25904aeb79f7337d28880e254741e2bab9b"}
Jul 07 02:18:13.052297 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:13.052238    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-101.ec2.internal" podStartSLOduration=4.052216241 podStartE2EDuration="4.052216241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:13.050585227 +0000 UTC m=+5.693745751" watchObservedRunningTime="2026-07-07 02:18:13.052216241 +0000 UTC m=+5.695376765"
Jul 07 02:18:13.052795 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:13.052762    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-101.ec2.internal" podStartSLOduration=4.052750308 podStartE2EDuration="4.052750308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:12.04412794 +0000 UTC m=+4.687288462" watchObservedRunningTime="2026-07-07 02:18:13.052750308 +0000 UTC m=+5.695910831"
Jul 07 02:18:13.437793 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:13.437706    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:13.437793 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:13.437789    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:13.438015 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:13.437918    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:13.438015 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:13.437980    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs podName:50534db7-9f97-4088-bc8a-b33de25d0d36 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:17.437963292 +0000 UTC m=+10.081123806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs") pod "network-metrics-daemon-wllmf" (UID: "50534db7-9f97-4088-bc8a-b33de25d0d36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:13.438426 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:13.438404    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:13.438518 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:13.438460    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret podName:dfd04d15-d522-4e36-83ff-1631d6f859aa nodeName:}" failed. No retries permitted until 2026-07-07 02:18:17.438446634 +0000 UTC m=+10.081607139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret") pod "global-pull-secret-syncer-drrrz" (UID: "dfd04d15-d522-4e36-83ff-1631d6f859aa") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:13.538775 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:13.538736    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdjb\" (UniqueName: \"kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb\") pod \"network-check-target-6kx2t\" (UID: \"47733a2c-fd25-4bfe-9883-84726a7029d8\") " pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:13.539001 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:13.538983    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:13.539077 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:13.539012    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:13.539077 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:13.539025    2543 projected.go:196] Error preparing data for projected volume kube-api-access-cqdjb for pod openshift-network-diagnostics/network-check-target-6kx2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:13.539211 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:13.539083    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb podName:47733a2c-fd25-4bfe-9883-84726a7029d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:17.539066068 +0000 UTC m=+10.182226571 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqdjb" (UniqueName: "kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb") pod "network-check-target-6kx2t" (UID: "47733a2c-fd25-4bfe-9883-84726a7029d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:13.976447 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:13.976357    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:13.976965 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:13.976488    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:14.974948 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:14.974917    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:14.975116 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:14.975025    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:14.975369 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:14.975352    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:14.975466 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:14.975450    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:15.974317 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:15.974289    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:15.974662 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:15.974418    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:16.975330 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:16.974803    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:16.975330 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:16.974899    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:16.975330 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:16.975207    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:16.975330 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:16.975283    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:17.473931 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:17.473339    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:17.473931 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:17.473415    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:17.473931 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:17.473464    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:17.473931 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:17.473504    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:17.473931 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:17.473528    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret podName:dfd04d15-d522-4e36-83ff-1631d6f859aa nodeName:}" failed. No retries permitted until 2026-07-07 02:18:25.473511182 +0000 UTC m=+18.116671705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret") pod "global-pull-secret-syncer-drrrz" (UID: "dfd04d15-d522-4e36-83ff-1631d6f859aa") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:17.473931 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:17.473546    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs podName:50534db7-9f97-4088-bc8a-b33de25d0d36 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:25.473537069 +0000 UTC m=+18.116697570 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs") pod "network-metrics-daemon-wllmf" (UID: "50534db7-9f97-4088-bc8a-b33de25d0d36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:17.574305 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:17.574266    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdjb\" (UniqueName: \"kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb\") pod \"network-check-target-6kx2t\" (UID: \"47733a2c-fd25-4bfe-9883-84726a7029d8\") " pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:17.574475 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:17.574462    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:17.574551 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:17.574488    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:17.574551 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:17.574502    2543 projected.go:196] Error preparing data for projected volume kube-api-access-cqdjb for pod openshift-network-diagnostics/network-check-target-6kx2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:17.574646 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:17.574564    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb podName:47733a2c-fd25-4bfe-9883-84726a7029d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:25.574542364 +0000 UTC m=+18.217702884 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqdjb" (UniqueName: "kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb") pod "network-check-target-6kx2t" (UID: "47733a2c-fd25-4bfe-9883-84726a7029d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:17.975005 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:17.974973    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:17.975217 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:17.975140    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:18.974816 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:18.974330    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:18.974816 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:18.974372    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:18.974816 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:18.974465    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:18.974816 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:18.974570    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:19.974383 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:19.974348    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:19.974562 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:19.974471    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:20.974619 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:20.974581    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:20.975078 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:20.974594    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:20.975078 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:20.974705    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:20.975078 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:20.974820    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:21.974780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:21.974748    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:21.975237 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:21.974882    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:22.974198 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:22.974160    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:22.974369 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:22.974161    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:22.974369 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:22.974325    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:22.974476 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:22.974393    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:23.974453 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:23.974419    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:23.974903 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:23.974544    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:24.974680 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:24.974641    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:24.975207 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:24.974641    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:24.975207 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:24.974801    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:24.975207 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:24.974846    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:25.535744 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:25.535712    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:25.535947 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:25.535774    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:25.535947 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:25.535846    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:25.535947 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:25.535858    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:25.535947 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:25.535914    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret podName:dfd04d15-d522-4e36-83ff-1631d6f859aa nodeName:}" failed. No retries permitted until 2026-07-07 02:18:41.535894646 +0000 UTC m=+34.179055160 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret") pod "global-pull-secret-syncer-drrrz" (UID: "dfd04d15-d522-4e36-83ff-1631d6f859aa") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:25.535947 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:25.535932    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs podName:50534db7-9f97-4088-bc8a-b33de25d0d36 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:41.535923447 +0000 UTC m=+34.179083954 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs") pod "network-metrics-daemon-wllmf" (UID: "50534db7-9f97-4088-bc8a-b33de25d0d36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:25.636820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:25.636782    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdjb\" (UniqueName: \"kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb\") pod \"network-check-target-6kx2t\" (UID: \"47733a2c-fd25-4bfe-9883-84726a7029d8\") " pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:25.637015 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:25.636962    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:25.637015 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:25.636983    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:25.637015 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:25.636992    2543 projected.go:196] Error preparing data for projected volume kube-api-access-cqdjb for pod openshift-network-diagnostics/network-check-target-6kx2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:25.637150 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:25.637047    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb podName:47733a2c-fd25-4bfe-9883-84726a7029d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:41.637028526 +0000 UTC m=+34.280189038 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqdjb" (UniqueName: "kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb") pod "network-check-target-6kx2t" (UID: "47733a2c-fd25-4bfe-9883-84726a7029d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:25.974530 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:25.974444    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:25.974685 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:25.974560    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:26.974180 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:26.974146    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:26.974612 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:26.974196    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:26.974612 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:26.974301    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:26.974612 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:26.974429    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:27.975294 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:27.975255    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:27.975717 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:27.975371    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:28.974517 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:28.974478    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:28.974711 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:28.974630    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:28.974711 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:28.974668    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:28.974820 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:28.974736    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:29.974360 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:29.974236    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:29.974961 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:29.974465    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:30.073962 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.073762    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vkx8c" event={"ID":"6fb4b23e-61ae-449e-abad-c58f2c9f5bc6","Type":"ContainerStarted","Data":"bdab523ee9c2f3bf71e0423cb63558353b695f7504393af4acab8cb9befd3b31"}
Jul 07 02:18:30.075228 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.075178    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm" event={"ID":"8840d1c9-665e-45fb-b8f3-1bf632053a01","Type":"ContainerStarted","Data":"5d54bf7893e68db1e570ff6bc4f129eeb04cbd848fa6e99d16eea30ea480fa9c"}
Jul 07 02:18:30.079019 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.078993    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m7vbn" event={"ID":"5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e","Type":"ContainerStarted","Data":"1e017d26c44e617d69f293763b09b052e1e2d2147923bc51ab7824a19d85be50"}
Jul 07 02:18:30.081587 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.081563    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp" event={"ID":"de69c094-35c2-4b5a-8d70-efdc95adec5b","Type":"ContainerStarted","Data":"683f530d0c504bd4249c58b930488fdd2c6842b7e8b3b1f9bf3e20b908bad1f5"}
Jul 07 02:18:30.081688 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.081627    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp" event={"ID":"de69c094-35c2-4b5a-8d70-efdc95adec5b","Type":"ContainerStarted","Data":"5225d3010c704cf852d40fd1557cbab97c7b362978b9c4f336713ff26c5b9fdf"}
Jul 07 02:18:30.081688 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.081659    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp" event={"ID":"de69c094-35c2-4b5a-8d70-efdc95adec5b","Type":"ContainerStarted","Data":"b4bfb3babbda0703392fef8d61c21233f6c85ee7b893c609fd8077b13179d3b4"}
Jul 07 02:18:30.082918 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.082894    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nqvp2" event={"ID":"e052eaa2-eca7-4894-8be6-e6b273851fa7","Type":"ContainerStarted","Data":"0dbd678751822f7287c4ab79a2ab023f039fea68cea477f842ca6d73ed83b49a"}
Jul 07 02:18:30.084322 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.084294    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6l9zx" event={"ID":"c703c034-4b57-4d22-a865-01ab46b75790","Type":"ContainerStarted","Data":"13e9731e6b303555fea331ae6a734c03804c67d6e80d421a292758c0f069d1a0"}
Jul 07 02:18:30.085521 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.085504    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6tqbv" event={"ID":"910a95ac-f4c9-42e9-ae74-c22ed25c6be0","Type":"ContainerStarted","Data":"a935677d732c969e283f5244b5622c1b9d90a067a55205727a32490a9251792e"}
Jul 07 02:18:30.086791 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.086771    2543 generic.go:356] "Generic (PLEG): container finished" podID="bca64acd-88bf-4c2e-9677-50add32ca830" containerID="0bc051cdf57e792d22fb2711ba22d056d3e417fdfcc582721f90ecee5c671993" exitCode=0
Jul 07 02:18:30.086886 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.086797    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdd9x" event={"ID":"bca64acd-88bf-4c2e-9677-50add32ca830","Type":"ContainerDied","Data":"0bc051cdf57e792d22fb2711ba22d056d3e417fdfcc582721f90ecee5c671993"}
Jul 07 02:18:30.087501 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.087445    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vkx8c" podStartSLOduration=8.899439667 podStartE2EDuration="23.087428812s" totalImagesPullingTime="14.187989145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.087275837 +0000 UTC m=+22.730436361" watchObservedRunningTime="2026-07-07 02:18:30.087428812 +0000 UTC m=+22.730589335"
Jul 07 02:18:30.123684 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.123642    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m7vbn" podStartSLOduration=3.471222284 podStartE2EDuration="22.12362688s" totalImagesPullingTime="18.652404596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.123614558 +0000 UTC m=+22.766775095" watchObservedRunningTime="2026-07-07 02:18:30.12362688 +0000 UTC m=+22.766787402"
Jul 07 02:18:30.137621 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.137562    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nqvp2" podStartSLOduration=3.5396591969999998 podStartE2EDuration="22.137540762s" totalImagesPullingTime="18.597881565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.136396217 +0000 UTC m=+22.779556741" watchObservedRunningTime="2026-07-07 02:18:30.137540762 +0000 UTC m=+22.780701335"
Jul 07 02:18:30.154534 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.154486    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6tqbv" podStartSLOduration=3.511343029 podStartE2EDuration="22.154471483s" totalImagesPullingTime="18.643128454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.153766261 +0000 UTC m=+22.796926783" watchObservedRunningTime="2026-07-07 02:18:30.154471483 +0000 UTC m=+22.797632005"
Jul 07 02:18:30.167401 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.167348    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6l9zx" podStartSLOduration=7.968678177 podStartE2EDuration="22.167327919s" totalImagesPullingTime="14.198649742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:30.167078677 +0000 UTC m=+22.810239199" watchObservedRunningTime="2026-07-07 02:18:30.167327919 +0000 UTC m=+22.810488442"
Jul 07 02:18:30.974236 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.974208    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:30.974344 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:30.974253    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:30.974382 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:30.974350    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:30.974681 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:30.974462    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:31.092825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:31.092792    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp" event={"ID":"de69c094-35c2-4b5a-8d70-efdc95adec5b","Type":"ContainerStarted","Data":"17bf7ee2054c47e04b975ceeb57e22498d71f0133f1f82ff01f547ef80040253"}
Jul 07 02:18:31.092985 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:31.092897    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp" event={"ID":"de69c094-35c2-4b5a-8d70-efdc95adec5b","Type":"ContainerStarted","Data":"9aeeb98e5b877240dce728acc8a4b34d857b5d9406a6828b01f55112be06d3da"}
Jul 07 02:18:31.092985 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:31.092944    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp" event={"ID":"de69c094-35c2-4b5a-8d70-efdc95adec5b","Type":"ContainerStarted","Data":"793892a234050e2e2f0335cadf7bf4b4dd699c308bb24e92c01f01edacd75441"}
Jul 07 02:18:31.094585 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:31.094540    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tqlvt" event={"ID":"352d6441-d9b5-49c7-a3b3-e249addf652b","Type":"ContainerStarted","Data":"adf99425f2befca2bc6c65eef968da2e9524786e380691e2b29d2aaa7f6d8b86"}
Jul 07 02:18:31.113493 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:31.113469    2543 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 07 02:18:31.114708 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:31.114660    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tqlvt" podStartSLOduration=4.510741481 podStartE2EDuration="23.114641635s" totalImagesPullingTime="18.603900154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:31.114315051 +0000 UTC m=+23.757475590" watchObservedRunningTime="2026-07-07 02:18:31.114641635 +0000 UTC m=+23.757802159"
Jul 07 02:18:31.881292 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:31.881151    2543 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-07T02:18:31.113486874Z","UUID":"28f12965-8c6e-4f5f-8a37-33d034e02acd","Handler":null,"Name":"","Endpoint":""}
Jul 07 02:18:31.882820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:31.882800    2543 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 07 02:18:31.882951 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:31.882837    2543 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 07 02:18:31.974336 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:31.974310    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:31.974562 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:31.974419    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:32.098992 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:32.098954    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm" event={"ID":"8840d1c9-665e-45fb-b8f3-1bf632053a01","Type":"ContainerStarted","Data":"363982ade3ea1fa8d81915a99a96ebdfd5a4db12d34d435db3dd858796a3d9aa"}
Jul 07 02:18:32.730128 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:32.730052    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:32.730750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:32.730733    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:32.974340 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:32.974128    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:32.974591 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:32.974201    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:32.974591 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:32.974422    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:32.974591 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:32.974519    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:33.105141 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:33.105063    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp" event={"ID":"de69c094-35c2-4b5a-8d70-efdc95adec5b","Type":"ContainerStarted","Data":"d89173e1f6924412e3f5fca552d637ad28312e371518e041afe612df83704566"}
Jul 07 02:18:33.107396 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:33.107363    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm" event={"ID":"8840d1c9-665e-45fb-b8f3-1bf632053a01","Type":"ContainerStarted","Data":"d4f0a4cecea2ec51e05bfb8f4d80f75b53a436dabf08a49c056c17b7963c325d"}
Jul 07 02:18:33.107602 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:33.107581    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:33.108358 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:33.108339    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vkx8c"
Jul 07 02:18:33.129855 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:33.129787    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2cmtm" podStartSLOduration=3.974432804 podStartE2EDuration="25.129768407s" totalImagesPullingTime="21.155335603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:33.129216812 +0000 UTC m=+25.772377334" watchObservedRunningTime="2026-07-07 02:18:33.129768407 +0000 UTC m=+25.772928930"
Jul 07 02:18:33.978816 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:33.978790    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:33.979242 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:33.978897    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:34.974656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:34.974468    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:34.974825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:34.974514    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:34.974825 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:34.974726    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:34.974825 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:34.974806    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:35.113753 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:35.113720    2543 generic.go:356] "Generic (PLEG): container finished" podID="bca64acd-88bf-4c2e-9677-50add32ca830" containerID="ec8320a5c4a21f1ffbe561fae1a9b38858c87358804d4202c6882937f274fb43" exitCode=0
Jul 07 02:18:35.114461 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:35.113809    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdd9x" event={"ID":"bca64acd-88bf-4c2e-9677-50add32ca830","Type":"ContainerDied","Data":"ec8320a5c4a21f1ffbe561fae1a9b38858c87358804d4202c6882937f274fb43"}
Jul 07 02:18:35.117864 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:35.117837    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp" event={"ID":"de69c094-35c2-4b5a-8d70-efdc95adec5b","Type":"ContainerStarted","Data":"7773644e2bcaf418861ac586abc2346b9ad06f0331c969a4d190fc41fe9ed2f9"}
Jul 07 02:18:35.173691 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:35.173600    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp" podStartSLOduration=9.498324687 podStartE2EDuration="28.173585118s" totalImagesPullingTime="18.675260431s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:35.173210576 +0000 UTC m=+27.816371098" watchObservedRunningTime="2026-07-07 02:18:35.173585118 +0000 UTC m=+27.816745640"
Jul 07 02:18:35.979750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:35.979240    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:35.979750 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:35.979362    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:36.120222 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.120198    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:36.120657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.120324    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:36.120657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.120380    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:36.136481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.136460    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:36.139656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.139633    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:18:36.652377 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.652298    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6kx2t"]
Jul 07 02:18:36.652626 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.652412    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:36.652626 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:36.652540    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:36.655792 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.655764    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-drrrz"]
Jul 07 02:18:36.655959 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.655864    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:36.655959 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:36.655935    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:36.656364 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.656345    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wllmf"]
Jul 07 02:18:36.656460 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:36.656442    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:36.656583 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:36.656541    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:37.123787 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:37.123702    2543 generic.go:356] "Generic (PLEG): container finished" podID="bca64acd-88bf-4c2e-9677-50add32ca830" containerID="39593a519a191f87020c8743d55c972b4b6e2cfa67c8ebbbb4a4ba9284102044" exitCode=0
Jul 07 02:18:37.124356 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:37.123798    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdd9x" event={"ID":"bca64acd-88bf-4c2e-9677-50add32ca830","Type":"ContainerDied","Data":"39593a519a191f87020c8743d55c972b4b6e2cfa67c8ebbbb4a4ba9284102044"}
Jul 07 02:18:37.977929 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:37.977765    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:37.978094 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:37.977819    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:37.978094 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:37.977991    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:37.978094 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:37.977819    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:37.978094 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:37.978074    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:37.978272 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:37.978130    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:39.130461 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:39.130382    2543 generic.go:356] "Generic (PLEG): container finished" podID="bca64acd-88bf-4c2e-9677-50add32ca830" containerID="127b03ca59cffb19209463754c62613af2a06867d77ebbea59085d62a7229fe6" exitCode=0
Jul 07 02:18:39.130461 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:39.130448    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdd9x" event={"ID":"bca64acd-88bf-4c2e-9677-50add32ca830","Type":"ContainerDied","Data":"127b03ca59cffb19209463754c62613af2a06867d77ebbea59085d62a7229fe6"}
Jul 07 02:18:39.978053 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:39.978018    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:39.978053 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:39.978039    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:39.978053 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:39.978055    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:39.978436 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:39.978146    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:39.978436 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:39.978266    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:39.978436 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:39.978336    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:41.559320 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:41.559279    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:41.559757 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:41.559337    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:41.559757 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.559422    2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:41.559757 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.559448    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:41.559757 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.559488    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret podName:dfd04d15-d522-4e36-83ff-1631d6f859aa nodeName:}" failed. No retries permitted until 2026-07-07 02:19:13.559471935 +0000 UTC m=+66.202632436 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret") pod "global-pull-secret-syncer-drrrz" (UID: "dfd04d15-d522-4e36-83ff-1631d6f859aa") : object "kube-system"/"original-pull-secret" not registered
Jul 07 02:18:41.559757 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.559502    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs podName:50534db7-9f97-4088-bc8a-b33de25d0d36 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:13.559495492 +0000 UTC m=+66.202655992 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs") pod "network-metrics-daemon-wllmf" (UID: "50534db7-9f97-4088-bc8a-b33de25d0d36") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 07 02:18:41.660364 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:41.660328    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdjb\" (UniqueName: \"kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb\") pod \"network-check-target-6kx2t\" (UID: \"47733a2c-fd25-4bfe-9883-84726a7029d8\") " pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:41.660550 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.660507    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 07 02:18:41.660550 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.660532    2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 07 02:18:41.660550 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.660545    2543 projected.go:196] Error preparing data for projected volume kube-api-access-cqdjb for pod openshift-network-diagnostics/network-check-target-6kx2t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:41.660671 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.660604    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb podName:47733a2c-fd25-4bfe-9883-84726a7029d8 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:13.660587893 +0000 UTC m=+66.303748401 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cqdjb" (UniqueName: "kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb") pod "network-check-target-6kx2t" (UID: "47733a2c-fd25-4bfe-9883-84726a7029d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 07 02:18:41.974606 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:41.974533    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:41.974757 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:41.974533    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:41.974757 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.974650    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drrrz" podUID="dfd04d15-d522-4e36-83ff-1631d6f859aa"
Jul 07 02:18:41.974757 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:41.974543    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:41.974757 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.974721    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-6kx2t" podUID="47733a2c-fd25-4bfe-9883-84726a7029d8"
Jul 07 02:18:41.974944 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:41.974803    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:18:42.601940 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.601859    2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-101.ec2.internal" event="NodeReady"
Jul 07 02:18:42.602505 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.602015    2543 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 07 02:18:42.654887 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.654837    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dzjkl"]
Jul 07 02:18:42.689920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.689888    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5hrxw"]
Jul 07 02:18:42.690113 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.690052    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.692515 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.692493    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 07 02:18:42.692731 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.692715    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l8w8v\""
Jul 07 02:18:42.692950 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.692936    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 07 02:18:42.703713 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.703693    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5hrxw"]
Jul 07 02:18:42.703820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.703727    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dzjkl"]
Jul 07 02:18:42.703820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.703812    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5hrxw"
Jul 07 02:18:42.706433 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.706413    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 07 02:18:42.706433 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.706423    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pm56q\""
Jul 07 02:18:42.706433 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.706429    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 07 02:18:42.706657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.706471    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 07 02:18:42.770258 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.770222    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2lj\" (UniqueName: \"kubernetes.io/projected/5921583f-ac66-4059-80f0-0cf81c329d20-kube-api-access-8q2lj\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.770446 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.770278    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.770446 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.770318    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5921583f-ac66-4059-80f0-0cf81c329d20-tmp-dir\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.770446 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.770368    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5921583f-ac66-4059-80f0-0cf81c329d20-config-volume\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.871647 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.871554    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2lj\" (UniqueName: \"kubernetes.io/projected/5921583f-ac66-4059-80f0-0cf81c329d20-kube-api-access-8q2lj\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.872298 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.872007    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.872298 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.872071    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g88gp\" (UniqueName: \"kubernetes.io/projected/25136846-eadf-4a17-93c9-76e0191971ff-kube-api-access-g88gp\") pod \"ingress-canary-5hrxw\" (UID: \"25136846-eadf-4a17-93c9-76e0191971ff\") " pod="openshift-ingress-canary/ingress-canary-5hrxw"
Jul 07 02:18:42.872298 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:42.872259    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:42.872574 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:42.872329    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls podName:5921583f-ac66-4059-80f0-0cf81c329d20 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:43.37230757 +0000 UTC m=+36.015468092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls") pod "dns-default-dzjkl" (UID: "5921583f-ac66-4059-80f0-0cf81c329d20") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:42.872574 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.872351    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5921583f-ac66-4059-80f0-0cf81c329d20-tmp-dir\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.872574 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.872403    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25136846-eadf-4a17-93c9-76e0191971ff-cert\") pod \"ingress-canary-5hrxw\" (UID: \"25136846-eadf-4a17-93c9-76e0191971ff\") " pod="openshift-ingress-canary/ingress-canary-5hrxw"
Jul 07 02:18:42.872574 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.872439    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5921583f-ac66-4059-80f0-0cf81c329d20-config-volume\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.872792 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.872691    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5921583f-ac66-4059-80f0-0cf81c329d20-tmp-dir\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.874628 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.874594    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5921583f-ac66-4059-80f0-0cf81c329d20-config-volume\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.882900 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.882871    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2lj\" (UniqueName: \"kubernetes.io/projected/5921583f-ac66-4059-80f0-0cf81c329d20-kube-api-access-8q2lj\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:42.973752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.973710    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g88gp\" (UniqueName: \"kubernetes.io/projected/25136846-eadf-4a17-93c9-76e0191971ff-kube-api-access-g88gp\") pod \"ingress-canary-5hrxw\" (UID: \"25136846-eadf-4a17-93c9-76e0191971ff\") " pod="openshift-ingress-canary/ingress-canary-5hrxw"
Jul 07 02:18:42.973931 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.973784    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25136846-eadf-4a17-93c9-76e0191971ff-cert\") pod \"ingress-canary-5hrxw\" (UID: \"25136846-eadf-4a17-93c9-76e0191971ff\") " pod="openshift-ingress-canary/ingress-canary-5hrxw"
Jul 07 02:18:42.976626 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.976601    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/25136846-eadf-4a17-93c9-76e0191971ff-cert\") pod \"ingress-canary-5hrxw\" (UID: \"25136846-eadf-4a17-93c9-76e0191971ff\") " pod="openshift-ingress-canary/ingress-canary-5hrxw"
Jul 07 02:18:42.982923 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:42.982888    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g88gp\" (UniqueName: \"kubernetes.io/projected/25136846-eadf-4a17-93c9-76e0191971ff-kube-api-access-g88gp\") pod \"ingress-canary-5hrxw\" (UID: \"25136846-eadf-4a17-93c9-76e0191971ff\") " pod="openshift-ingress-canary/ingress-canary-5hrxw"
Jul 07 02:18:43.014710 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.014669    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5hrxw"
Jul 07 02:18:43.346503 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.346260    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5hrxw"]
Jul 07 02:18:43.376400 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.376364    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:43.376547 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:43.376493    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:43.376612 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:43.376564    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls podName:5921583f-ac66-4059-80f0-0cf81c329d20 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:44.376543932 +0000 UTC m=+37.019704449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls") pod "dns-default-dzjkl" (UID: "5921583f-ac66-4059-80f0-0cf81c329d20") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:43.411692 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:18:43.411661    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25136846_eadf_4a17_93c9_76e0191971ff.slice/crio-76996ec43e22a670882dc9f637e238db2e9fa44c9210c79f54dafbb063f45ffc WatchSource:0}: Error finding container 76996ec43e22a670882dc9f637e238db2e9fa44c9210c79f54dafbb063f45ffc: Status 404 returned error can't find the container with id 76996ec43e22a670882dc9f637e238db2e9fa44c9210c79f54dafbb063f45ffc
Jul 07 02:18:43.974504 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.974464    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:18:43.975007 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.974464    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:18:43.975007 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.974466    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:18:43.977382 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.977361    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 02:18:43.978300 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.978279    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 02:18:43.978300 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.978297    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v4j68\""
Jul 07 02:18:43.978457 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.978283    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwgqf\""
Jul 07 02:18:43.978457 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.978331    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 02:18:43.978457 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:43.978285    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 02:18:44.144156 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:44.144121    2543 generic.go:356] "Generic (PLEG): container finished" podID="bca64acd-88bf-4c2e-9677-50add32ca830" containerID="6a669bf9fdbb4c446758be07e30d72bcf52f0afa853b00ff2e97f4c3842177f5" exitCode=0
Jul 07 02:18:44.144336 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:44.144219    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdd9x" event={"ID":"bca64acd-88bf-4c2e-9677-50add32ca830","Type":"ContainerDied","Data":"6a669bf9fdbb4c446758be07e30d72bcf52f0afa853b00ff2e97f4c3842177f5"}
Jul 07 02:18:44.145445 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:44.145420    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5hrxw" event={"ID":"25136846-eadf-4a17-93c9-76e0191971ff","Type":"ContainerStarted","Data":"76996ec43e22a670882dc9f637e238db2e9fa44c9210c79f54dafbb063f45ffc"}
Jul 07 02:18:44.383992 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:44.383957    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:44.384168 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:44.384083    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:44.384168 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:44.384148    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls podName:5921583f-ac66-4059-80f0-0cf81c329d20 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:46.384130276 +0000 UTC m=+39.027290777 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls") pod "dns-default-dzjkl" (UID: "5921583f-ac66-4059-80f0-0cf81c329d20") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:45.151382 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:45.151339    2543 generic.go:356] "Generic (PLEG): container finished" podID="bca64acd-88bf-4c2e-9677-50add32ca830" containerID="9e966f1be87b2851b682a8015a28657fa58ec9a6db9d167f783dcec303b7e59f" exitCode=0
Jul 07 02:18:45.151913 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:45.151390    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdd9x" event={"ID":"bca64acd-88bf-4c2e-9677-50add32ca830","Type":"ContainerDied","Data":"9e966f1be87b2851b682a8015a28657fa58ec9a6db9d167f783dcec303b7e59f"}
Jul 07 02:18:46.157266 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:46.157050    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sdd9x" event={"ID":"bca64acd-88bf-4c2e-9677-50add32ca830","Type":"ContainerStarted","Data":"a776a4c8c6ccca8fafbf9db01f7d2980c4f6c7a895b15efa99d308dd8ea88e95"}
Jul 07 02:18:46.158498 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:46.158477    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5hrxw" event={"ID":"25136846-eadf-4a17-93c9-76e0191971ff","Type":"ContainerStarted","Data":"2d5554f2868f1d058b2b5cb8c5d31cac0aed3ebdc3a01d28c830485bbb5fc06b"}
Jul 07 02:18:46.180648 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:46.180602    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sdd9x" podStartSLOduration=8.492248875 podStartE2EDuration="38.180587696s" totalImagesPullingTime="29.688338821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:46.179156113 +0000 UTC m=+38.822316615" watchObservedRunningTime="2026-07-07 02:18:46.180587696 +0000 UTC m=+38.823748218"
Jul 07 02:18:46.194117 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:46.194066    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5hrxw" podStartSLOduration=2.111449689 podStartE2EDuration="4.194048041s" totalImagesPullingTime="2.082598352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:18:46.193253578 +0000 UTC m=+38.836414101" watchObservedRunningTime="2026-07-07 02:18:46.194048041 +0000 UTC m=+38.837208565"
Jul 07 02:18:46.400973 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:46.400922    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:46.401124 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:46.401048    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:46.401124 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:46.401100    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls podName:5921583f-ac66-4059-80f0-0cf81c329d20 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:50.401087315 +0000 UTC m=+43.044247816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls") pod "dns-default-dzjkl" (UID: "5921583f-ac66-4059-80f0-0cf81c329d20") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:50.430911 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:50.430871    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:50.431326 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:50.431049    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:50.431326 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:50.431125    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls podName:5921583f-ac66-4059-80f0-0cf81c329d20 nodeName:}" failed. No retries permitted until 2026-07-07 02:18:58.431106755 +0000 UTC m=+51.074267257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls") pod "dns-default-dzjkl" (UID: "5921583f-ac66-4059-80f0-0cf81c329d20") : secret "dns-default-metrics-tls" not found
Jul 07 02:18:58.486063 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:18:58.486022    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:18:58.486574 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:58.486198    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:18:58.486574 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:18:58.486263    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls podName:5921583f-ac66-4059-80f0-0cf81c329d20 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:14.486247383 +0000 UTC m=+67.129407883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls") pod "dns-default-dzjkl" (UID: "5921583f-ac66-4059-80f0-0cf81c329d20") : secret "dns-default-metrics-tls" not found
Jul 07 02:19:08.136146 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:08.136121    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gh2mp"
Jul 07 02:19:11.108052 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.108012    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"]
Jul 07 02:19:11.110657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.110639    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"
Jul 07 02:19:11.114236 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.114208    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jrw5j\""
Jul 07 02:19:11.114236 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.114208    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 07 02:19:11.114419 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.114246    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 07 02:19:11.114419 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.114220    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 07 02:19:11.114419 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.114212    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 07 02:19:11.121242 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.121222    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"]
Jul 07 02:19:11.146942 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.146915    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"]
Jul 07 02:19:11.148547 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.148532    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.151066 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.151050    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 07 02:19:11.163523 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.161580    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"]
Jul 07 02:19:11.171338 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.171313    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j8z8\" (UniqueName: \"kubernetes.io/projected/a36926b9-cd6e-4def-b66a-ee485739af82-kube-api-access-5j8z8\") pod \"managed-serviceaccount-addon-agent-744f7bf9c8-br8j6\" (UID: \"a36926b9-cd6e-4def-b66a-ee485739af82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"
Jul 07 02:19:11.171454 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.171343    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e3999a8-8f72-47ad-9479-226f7b4b7119-tmp\") pod \"klusterlet-addon-workmgr-6d48ccb978-jst97\" (UID: \"7e3999a8-8f72-47ad-9479-226f7b4b7119\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.171454 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.171361    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7e3999a8-8f72-47ad-9479-226f7b4b7119-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d48ccb978-jst97\" (UID: \"7e3999a8-8f72-47ad-9479-226f7b4b7119\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.171454 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.171397    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccmsr\" (UniqueName: \"kubernetes.io/projected/7e3999a8-8f72-47ad-9479-226f7b4b7119-kube-api-access-ccmsr\") pod \"klusterlet-addon-workmgr-6d48ccb978-jst97\" (UID: \"7e3999a8-8f72-47ad-9479-226f7b4b7119\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.171594 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.171473    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36926b9-cd6e-4def-b66a-ee485739af82-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-744f7bf9c8-br8j6\" (UID: \"a36926b9-cd6e-4def-b66a-ee485739af82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"
Jul 07 02:19:11.271909 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.271864    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j8z8\" (UniqueName: \"kubernetes.io/projected/a36926b9-cd6e-4def-b66a-ee485739af82-kube-api-access-5j8z8\") pod \"managed-serviceaccount-addon-agent-744f7bf9c8-br8j6\" (UID: \"a36926b9-cd6e-4def-b66a-ee485739af82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"
Jul 07 02:19:11.271909 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.271909    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e3999a8-8f72-47ad-9479-226f7b4b7119-tmp\") pod \"klusterlet-addon-workmgr-6d48ccb978-jst97\" (UID: \"7e3999a8-8f72-47ad-9479-226f7b4b7119\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.272119 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.271936    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7e3999a8-8f72-47ad-9479-226f7b4b7119-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d48ccb978-jst97\" (UID: \"7e3999a8-8f72-47ad-9479-226f7b4b7119\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.272167 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.272141    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccmsr\" (UniqueName: \"kubernetes.io/projected/7e3999a8-8f72-47ad-9479-226f7b4b7119-kube-api-access-ccmsr\") pod \"klusterlet-addon-workmgr-6d48ccb978-jst97\" (UID: \"7e3999a8-8f72-47ad-9479-226f7b4b7119\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.272258 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.272242    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36926b9-cd6e-4def-b66a-ee485739af82-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-744f7bf9c8-br8j6\" (UID: \"a36926b9-cd6e-4def-b66a-ee485739af82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"
Jul 07 02:19:11.272329 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.272309    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7e3999a8-8f72-47ad-9479-226f7b4b7119-tmp\") pod \"klusterlet-addon-workmgr-6d48ccb978-jst97\" (UID: \"7e3999a8-8f72-47ad-9479-226f7b4b7119\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.276088 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.276055    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/7e3999a8-8f72-47ad-9479-226f7b4b7119-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d48ccb978-jst97\" (UID: \"7e3999a8-8f72-47ad-9479-226f7b4b7119\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.276088 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.276085    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a36926b9-cd6e-4def-b66a-ee485739af82-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-744f7bf9c8-br8j6\" (UID: \"a36926b9-cd6e-4def-b66a-ee485739af82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"
Jul 07 02:19:11.280227 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.280202    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccmsr\" (UniqueName: \"kubernetes.io/projected/7e3999a8-8f72-47ad-9479-226f7b4b7119-kube-api-access-ccmsr\") pod \"klusterlet-addon-workmgr-6d48ccb978-jst97\" (UID: \"7e3999a8-8f72-47ad-9479-226f7b4b7119\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.280468 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.280444    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j8z8\" (UniqueName: \"kubernetes.io/projected/a36926b9-cd6e-4def-b66a-ee485739af82-kube-api-access-5j8z8\") pod \"managed-serviceaccount-addon-agent-744f7bf9c8-br8j6\" (UID: \"a36926b9-cd6e-4def-b66a-ee485739af82\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"
Jul 07 02:19:11.427954 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.427920    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"
Jul 07 02:19:11.457517 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.457484    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:11.582571 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.582538    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6"]
Jul 07 02:19:11.585707 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:19:11.585676    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda36926b9_cd6e_4def_b66a_ee485739af82.slice/crio-995e163ebf3aff0dd831729bd1ae9871275721e9c6684bf5e5a2eb30bf0f9488 WatchSource:0}: Error finding container 995e163ebf3aff0dd831729bd1ae9871275721e9c6684bf5e5a2eb30bf0f9488: Status 404 returned error can't find the container with id 995e163ebf3aff0dd831729bd1ae9871275721e9c6684bf5e5a2eb30bf0f9488
Jul 07 02:19:11.607312 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:11.607284    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"]
Jul 07 02:19:11.610329 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:19:11.610304    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e3999a8_8f72_47ad_9479_226f7b4b7119.slice/crio-9cf3c2335b5543ef28991a5aaf40da0e7887154d3576be625cc8e57a51f42c3b WatchSource:0}: Error finding container 9cf3c2335b5543ef28991a5aaf40da0e7887154d3576be625cc8e57a51f42c3b: Status 404 returned error can't find the container with id 9cf3c2335b5543ef28991a5aaf40da0e7887154d3576be625cc8e57a51f42c3b
Jul 07 02:19:12.225697 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:12.225661    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6" event={"ID":"a36926b9-cd6e-4def-b66a-ee485739af82","Type":"ContainerStarted","Data":"995e163ebf3aff0dd831729bd1ae9871275721e9c6684bf5e5a2eb30bf0f9488"}
Jul 07 02:19:12.226793 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:12.226776    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97" event={"ID":"7e3999a8-8f72-47ad-9479-226f7b4b7119","Type":"ContainerStarted","Data":"9cf3c2335b5543ef28991a5aaf40da0e7887154d3576be625cc8e57a51f42c3b"}
Jul 07 02:19:13.591359 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.591307    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:19:13.591827 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.591393    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:19:13.594527 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.594500    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 07 02:19:13.594737 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.594722    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 07 02:19:13.602672 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:19:13.602645    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 02:19:13.602816 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:19:13.602714    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs podName:50534db7-9f97-4088-bc8a-b33de25d0d36 nodeName:}" failed. No retries permitted until 2026-07-07 02:20:17.602697406 +0000 UTC m=+130.245857906 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs") pod "network-metrics-daemon-wllmf" (UID: "50534db7-9f97-4088-bc8a-b33de25d0d36") : secret "metrics-daemon-secret" not found
Jul 07 02:19:13.605250 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.605171    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd04d15-d522-4e36-83ff-1631d6f859aa-original-pull-secret\") pod \"global-pull-secret-syncer-drrrz\" (UID: \"dfd04d15-d522-4e36-83ff-1631d6f859aa\") " pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:19:13.692826 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.692783    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqdjb\" (UniqueName: \"kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb\") pod \"network-check-target-6kx2t\" (UID: \"47733a2c-fd25-4bfe-9883-84726a7029d8\") " pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:19:13.695736 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.695659    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 07 02:19:13.698813 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.698789    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drrrz"
Jul 07 02:19:13.705672 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.705649    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 07 02:19:13.716569 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.716543    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqdjb\" (UniqueName: \"kubernetes.io/projected/47733a2c-fd25-4bfe-9883-84726a7029d8-kube-api-access-cqdjb\") pod \"network-check-target-6kx2t\" (UID: \"47733a2c-fd25-4bfe-9883-84726a7029d8\") " pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:19:13.989821 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.989782    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v4j68\""
Jul 07 02:19:13.997434 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:13.997407    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:19:14.446825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:14.446797    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-drrrz"]
Jul 07 02:19:14.451572 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:19:14.451540    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd04d15_d522_4e36_83ff_1631d6f859aa.slice/crio-39c0f1bf0685ae409e78fc4cdd7acb124772c0ea217443b89224b75e64a8c07e WatchSource:0}: Error finding container 39c0f1bf0685ae409e78fc4cdd7acb124772c0ea217443b89224b75e64a8c07e: Status 404 returned error can't find the container with id 39c0f1bf0685ae409e78fc4cdd7acb124772c0ea217443b89224b75e64a8c07e
Jul 07 02:19:14.471343 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:14.471255    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-6kx2t"]
Jul 07 02:19:14.473854 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:19:14.473813    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47733a2c_fd25_4bfe_9883_84726a7029d8.slice/crio-98c8a3b75a87d6860a9f5784c20025dfd747da10085891a830746dcff5da0c27 WatchSource:0}: Error finding container 98c8a3b75a87d6860a9f5784c20025dfd747da10085891a830746dcff5da0c27: Status 404 returned error can't find the container with id 98c8a3b75a87d6860a9f5784c20025dfd747da10085891a830746dcff5da0c27
Jul 07 02:19:14.501149 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:14.501120    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:19:14.501310 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:19:14.501284    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:19:14.501578 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:19:14.501359    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls podName:5921583f-ac66-4059-80f0-0cf81c329d20 nodeName:}" failed. No retries permitted until 2026-07-07 02:19:46.501338876 +0000 UTC m=+99.144499397 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls") pod "dns-default-dzjkl" (UID: "5921583f-ac66-4059-80f0-0cf81c329d20") : secret "dns-default-metrics-tls" not found
Jul 07 02:19:15.237075 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:15.237040    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6" event={"ID":"a36926b9-cd6e-4def-b66a-ee485739af82","Type":"ContainerStarted","Data":"5d7878ff6158bc009cdae302349045a88f0331890be2880ad7bb86bd30195831"}
Jul 07 02:19:15.238366 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:15.238314    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-drrrz" event={"ID":"dfd04d15-d522-4e36-83ff-1631d6f859aa","Type":"ContainerStarted","Data":"39c0f1bf0685ae409e78fc4cdd7acb124772c0ea217443b89224b75e64a8c07e"}
Jul 07 02:19:15.239509 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:15.239486    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6kx2t" event={"ID":"47733a2c-fd25-4bfe-9883-84726a7029d8","Type":"ContainerStarted","Data":"98c8a3b75a87d6860a9f5784c20025dfd747da10085891a830746dcff5da0c27"}
Jul 07 02:19:15.253565 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:15.253485    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6" podStartSLOduration=1.499682553 podStartE2EDuration="4.253468618s" totalImagesPullingTime="2.753786065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:15.252731866 +0000 UTC m=+67.895892390" watchObservedRunningTime="2026-07-07 02:19:15.253468618 +0000 UTC m=+67.896629139"
Jul 07 02:19:16.244605 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:16.244564    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97" event={"ID":"7e3999a8-8f72-47ad-9479-226f7b4b7119","Type":"ContainerStarted","Data":"4df71bb617498e003ac1948796ac6b77fc5f257cc30c2e0af3c2a165cee41f6d"}
Jul 07 02:19:16.260882 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:16.260821    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97" podStartSLOduration=1.206127468 podStartE2EDuration="5.260807089s" totalImagesPullingTime="4.054679621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:19:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:16.259634271 +0000 UTC m=+68.902794793" watchObservedRunningTime="2026-07-07 02:19:16.260807089 +0000 UTC m=+68.903967611"
Jul 07 02:19:17.248093 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:17.248061    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:17.249721 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:17.249687    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:19:20.256823 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:20.256729    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-drrrz" event={"ID":"dfd04d15-d522-4e36-83ff-1631d6f859aa","Type":"ContainerStarted","Data":"acb37115b2e352fa09939550bbd58169c93735835447209e5e1c85d172f56abf"}
Jul 07 02:19:20.258251 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:20.258229    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-6kx2t" event={"ID":"47733a2c-fd25-4bfe-9883-84726a7029d8","Type":"ContainerStarted","Data":"da869740457b196e1cf0ad522c509f8885ba347bd410150c5a1e4164022c6668"}
Jul 07 02:19:20.258360 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:20.258348    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:19:20.271938 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:20.271895    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-drrrz" podStartSLOduration=66.762020413 podStartE2EDuration="1m12.271881799s" totalImagesPullingTime="5.509861386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:20.271243565 +0000 UTC m=+72.914404089" watchObservedRunningTime="2026-07-07 02:19:20.271881799 +0000 UTC m=+72.915042320"
Jul 07 02:19:46.525680 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:46.525588    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:19:46.526094 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:19:46.525717    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:19:46.526094 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:19:46.525772    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls podName:5921583f-ac66-4059-80f0-0cf81c329d20 nodeName:}" failed. No retries permitted until 2026-07-07 02:20:50.525757561 +0000 UTC m=+163.168918061 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls") pod "dns-default-dzjkl" (UID: "5921583f-ac66-4059-80f0-0cf81c329d20") : secret "dns-default-metrics-tls" not found
Jul 07 02:19:51.262819 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:51.262788    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-6kx2t"
Jul 07 02:19:51.279786 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:19:51.279723    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-6kx2t" podStartSLOduration=97.794357515 podStartE2EDuration="1m43.279708779s" totalImagesPullingTime="5.485351264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:19:20.285767941 +0000 UTC m=+72.928928463" watchObservedRunningTime="2026-07-07 02:19:51.279708779 +0000 UTC m=+103.922869300"
Jul 07 02:20:17.650204 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:20:17.650137    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:20:17.650688 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:20:17.650293    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 02:20:17.650688 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:20:17.650361    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs podName:50534db7-9f97-4088-bc8a-b33de25d0d36 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:19.650344865 +0000 UTC m=+252.293505365 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs") pod "network-metrics-daemon-wllmf" (UID: "50534db7-9f97-4088-bc8a-b33de25d0d36") : secret "metrics-daemon-secret" not found
Jul 07 02:20:45.702752 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:20:45.702693    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dzjkl" podUID="5921583f-ac66-4059-80f0-0cf81c329d20"
Jul 07 02:20:46.460649 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:20:46.460621    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:20:46.993336 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:20:46.993289    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:20:50.576456 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:20:50.576412    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:20:50.576852 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:20:50.576534    2543 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 07 02:20:50.576852 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:20:50.576584    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls podName:5921583f-ac66-4059-80f0-0cf81c329d20 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:52.576572126 +0000 UTC m=+285.219732639 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls") pod "dns-default-dzjkl" (UID: "5921583f-ac66-4059-80f0-0cf81c329d20") : secret "dns-default-metrics-tls" not found
Jul 07 02:21:01.974530 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:01.974438    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:21:14.529622 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:14.529593    2543 generic.go:356] "Generic (PLEG): container finished" podID="a36926b9-cd6e-4def-b66a-ee485739af82" containerID="5d7878ff6158bc009cdae302349045a88f0331890be2880ad7bb86bd30195831" exitCode=255
Jul 07 02:21:14.529951 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:14.529630    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6" event={"ID":"a36926b9-cd6e-4def-b66a-ee485739af82","Type":"ContainerDied","Data":"5d7878ff6158bc009cdae302349045a88f0331890be2880ad7bb86bd30195831"}
Jul 07 02:21:14.529951 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:14.529885    2543 scope.go:117] "RemoveContainer" containerID="5d7878ff6158bc009cdae302349045a88f0331890be2880ad7bb86bd30195831"
Jul 07 02:21:15.534482 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:15.534443    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-744f7bf9c8-br8j6" event={"ID":"a36926b9-cd6e-4def-b66a-ee485739af82","Type":"ContainerStarted","Data":"7ec652367e7c44cf2599372527e92a0f1e3e94de0667214aa872a139569ae3cd"}
Jul 07 02:21:16.539722 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:16.539687    2543 generic.go:356] "Generic (PLEG): container finished" podID="7e3999a8-8f72-47ad-9479-226f7b4b7119" containerID="4df71bb617498e003ac1948796ac6b77fc5f257cc30c2e0af3c2a165cee41f6d" exitCode=1
Jul 07 02:21:16.540139 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:16.539732    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97" event={"ID":"7e3999a8-8f72-47ad-9479-226f7b4b7119","Type":"ContainerDied","Data":"4df71bb617498e003ac1948796ac6b77fc5f257cc30c2e0af3c2a165cee41f6d"}
Jul 07 02:21:16.540139 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:16.540022    2543 scope.go:117] "RemoveContainer" containerID="4df71bb617498e003ac1948796ac6b77fc5f257cc30c2e0af3c2a165cee41f6d"
Jul 07 02:21:17.248741 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:17.248691    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:21:17.543940 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:17.543858    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97" event={"ID":"7e3999a8-8f72-47ad-9479-226f7b4b7119","Type":"ContainerStarted","Data":"7de9f865178441e10836cc9e726b5bf4e14499f589e015d18702d58ec598f30d"}
Jul 07 02:21:17.544343 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:17.544052    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:21:17.544657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:21:17.544641    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d48ccb978-jst97"
Jul 07 02:22:05.398857 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.398823    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-chdqk"]
Jul 07 02:22:05.400787 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.400769    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.404089 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.404067    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 07 02:22:05.404219 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.404067    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:22:05.405155 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.405142    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 07 02:22:05.406946 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.406925    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tdnqd\""
Jul 07 02:22:05.407404 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.407230    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 07 02:22:05.412138 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.412119    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 07 02:22:05.423154 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.423123    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-chdqk"]
Jul 07 02:22:05.483017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.482984    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c294fd4-a99a-4441-9f26-b636c3b01562-config\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.483017 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.483020    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnrbq\" (UniqueName: \"kubernetes.io/projected/6c294fd4-a99a-4441-9f26-b636c3b01562-kube-api-access-wnrbq\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.483276 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.483073    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c294fd4-a99a-4441-9f26-b636c3b01562-trusted-ca\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.483276 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.483124    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c294fd4-a99a-4441-9f26-b636c3b01562-serving-cert\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.491459 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.491431    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw"]
Jul 07 02:22:05.493354 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.493335    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw"
Jul 07 02:22:05.496435 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.496418    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-j8fvl\""
Jul 07 02:22:05.504978 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.504957    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw"]
Jul 07 02:22:05.584286 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.584245    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c294fd4-a99a-4441-9f26-b636c3b01562-config\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.584286 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.584286    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnrbq\" (UniqueName: \"kubernetes.io/projected/6c294fd4-a99a-4441-9f26-b636c3b01562-kube-api-access-wnrbq\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.584521 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.584422    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c294fd4-a99a-4441-9f26-b636c3b01562-trusted-ca\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.584521 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.584453    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlcmj\" (UniqueName: \"kubernetes.io/projected/9f56cf3b-c21a-41be-8190-370b696e85bc-kube-api-access-rlcmj\") pod \"network-check-source-989f8b8dc-wpspw\" (UID: \"9f56cf3b-c21a-41be-8190-370b696e85bc\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw"
Jul 07 02:22:05.584521 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.584487    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c294fd4-a99a-4441-9f26-b636c3b01562-serving-cert\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.584956 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.584932    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c294fd4-a99a-4441-9f26-b636c3b01562-config\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.585178 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.585158    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6c294fd4-a99a-4441-9f26-b636c3b01562-trusted-ca\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.586820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.586801    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c294fd4-a99a-4441-9f26-b636c3b01562-serving-cert\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.593469 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.593446    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnrbq\" (UniqueName: \"kubernetes.io/projected/6c294fd4-a99a-4441-9f26-b636c3b01562-kube-api-access-wnrbq\") pod \"console-operator-68b757fb67-chdqk\" (UID: \"6c294fd4-a99a-4441-9f26-b636c3b01562\") " pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.685255 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.685157    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlcmj\" (UniqueName: \"kubernetes.io/projected/9f56cf3b-c21a-41be-8190-370b696e85bc-kube-api-access-rlcmj\") pod \"network-check-source-989f8b8dc-wpspw\" (UID: \"9f56cf3b-c21a-41be-8190-370b696e85bc\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw"
Jul 07 02:22:05.694041 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.694009    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlcmj\" (UniqueName: \"kubernetes.io/projected/9f56cf3b-c21a-41be-8190-370b696e85bc-kube-api-access-rlcmj\") pod \"network-check-source-989f8b8dc-wpspw\" (UID: \"9f56cf3b-c21a-41be-8190-370b696e85bc\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw"
Jul 07 02:22:05.709863 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.709836    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:05.801777 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.801745    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw"
Jul 07 02:22:05.831615 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.831582    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-chdqk"]
Jul 07 02:22:05.834964 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:22:05.834940    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c294fd4_a99a_4441_9f26_b636c3b01562.slice/crio-889f54145cb624b9b3c8b008c6b5ea70fdb00feafc2f07a985a1b7c99649188a WatchSource:0}: Error finding container 889f54145cb624b9b3c8b008c6b5ea70fdb00feafc2f07a985a1b7c99649188a: Status 404 returned error can't find the container with id 889f54145cb624b9b3c8b008c6b5ea70fdb00feafc2f07a985a1b7c99649188a
Jul 07 02:22:05.926256 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:05.926221    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw"]
Jul 07 02:22:05.929739 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:22:05.929720    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f56cf3b_c21a_41be_8190_370b696e85bc.slice/crio-08d211db44e4354fff4773626624dca27393dd3b29f00944b08c2647b36dfb63 WatchSource:0}: Error finding container 08d211db44e4354fff4773626624dca27393dd3b29f00944b08c2647b36dfb63: Status 404 returned error can't find the container with id 08d211db44e4354fff4773626624dca27393dd3b29f00944b08c2647b36dfb63
Jul 07 02:22:06.688002 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:06.687963    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw" event={"ID":"9f56cf3b-c21a-41be-8190-370b696e85bc","Type":"ContainerStarted","Data":"61f6509171ff8846cb1b1153ab8bd61e58f6bf6df42666d4a6d6b0ad2e25c5e9"}
Jul 07 02:22:06.688493 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:06.688022    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw" event={"ID":"9f56cf3b-c21a-41be-8190-370b696e85bc","Type":"ContainerStarted","Data":"08d211db44e4354fff4773626624dca27393dd3b29f00944b08c2647b36dfb63"}
Jul 07 02:22:06.689042 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:06.689020    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-chdqk" event={"ID":"6c294fd4-a99a-4441-9f26-b636c3b01562","Type":"ContainerStarted","Data":"889f54145cb624b9b3c8b008c6b5ea70fdb00feafc2f07a985a1b7c99649188a"}
Jul 07 02:22:06.703119 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:06.703076    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wpspw" podStartSLOduration=1.703061685 podStartE2EDuration="1.703061685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:06.702529554 +0000 UTC m=+239.345690080" watchObservedRunningTime="2026-07-07 02:22:06.703061685 +0000 UTC m=+239.346222201"
Jul 07 02:22:08.696944 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:08.696916    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/0.log"
Jul 07 02:22:08.697419 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:08.696956    2543 generic.go:356] "Generic (PLEG): container finished" podID="6c294fd4-a99a-4441-9f26-b636c3b01562" containerID="61fb1b3564de9f77aa40491d850693573cff45f85053922eaea3e96055a9243d" exitCode=255
Jul 07 02:22:08.697419 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:08.697023    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-chdqk" event={"ID":"6c294fd4-a99a-4441-9f26-b636c3b01562","Type":"ContainerDied","Data":"61fb1b3564de9f77aa40491d850693573cff45f85053922eaea3e96055a9243d"}
Jul 07 02:22:08.697419 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:08.697324    2543 scope.go:117] "RemoveContainer" containerID="61fb1b3564de9f77aa40491d850693573cff45f85053922eaea3e96055a9243d"
Jul 07 02:22:09.701155 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:09.701128    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:22:09.701568 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:09.701505    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/0.log"
Jul 07 02:22:09.701568 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:09.701540    2543 generic.go:356] "Generic (PLEG): container finished" podID="6c294fd4-a99a-4441-9f26-b636c3b01562" containerID="bb8909aeb3d5bfbc37e0ba49189632031f235be87dabf977e93f8fbb4359e66f" exitCode=255
Jul 07 02:22:09.701648 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:09.701569    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-chdqk" event={"ID":"6c294fd4-a99a-4441-9f26-b636c3b01562","Type":"ContainerDied","Data":"bb8909aeb3d5bfbc37e0ba49189632031f235be87dabf977e93f8fbb4359e66f"}
Jul 07 02:22:09.701648 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:09.701609    2543 scope.go:117] "RemoveContainer" containerID="61fb1b3564de9f77aa40491d850693573cff45f85053922eaea3e96055a9243d"
Jul 07 02:22:09.701835 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:09.701817    2543 scope.go:117] "RemoveContainer" containerID="bb8909aeb3d5bfbc37e0ba49189632031f235be87dabf977e93f8fbb4359e66f"
Jul 07 02:22:09.702010 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:09.701990    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-chdqk_openshift-console-operator(6c294fd4-a99a-4441-9f26-b636c3b01562)\"" pod="openshift-console-operator/console-operator-68b757fb67-chdqk" podUID="6c294fd4-a99a-4441-9f26-b636c3b01562"
Jul 07 02:22:10.466673 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.466642    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pkpzw"]
Jul 07 02:22:10.469124 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.469107    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.471540 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.471519    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 07 02:22:10.471654 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.471634    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 07 02:22:10.471874 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.471859    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 07 02:22:10.472040 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.472028    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6hsdg\""
Jul 07 02:22:10.472242 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.472225    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 07 02:22:10.486441 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.486310    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pkpzw"]
Jul 07 02:22:10.622002 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.621962    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-crio-socket\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.622002 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.622003    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.622269 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.622038    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtdc\" (UniqueName: \"kubernetes.io/projected/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-api-access-kwtdc\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.622269 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.622063    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-data-volume\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.622269 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.622138    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.705978 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.705954    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:22:10.706434 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.706289    2543 scope.go:117] "RemoveContainer" containerID="bb8909aeb3d5bfbc37e0ba49189632031f235be87dabf977e93f8fbb4359e66f"
Jul 07 02:22:10.706434 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:10.706423    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-chdqk_openshift-console-operator(6c294fd4-a99a-4441-9f26-b636c3b01562)\"" pod="openshift-console-operator/console-operator-68b757fb67-chdqk" podUID="6c294fd4-a99a-4441-9f26-b636c3b01562"
Jul 07 02:22:10.722799 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.722725    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtdc\" (UniqueName: \"kubernetes.io/projected/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-api-access-kwtdc\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.722799 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.722757    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-data-volume\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.722799 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.722788    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.722991 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:10.722841    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:22:10.722991 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:10.722896    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm podName:013a90cb-6fb8-4e1f-974d-04ea1d5078dc nodeName:}" failed. No retries permitted until 2026-07-07 02:22:11.222881218 +0000 UTC m=+243.866041742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkpzw" (UID: "013a90cb-6fb8-4e1f-974d-04ea1d5078dc") : configmap "kube-rbac-proxy" not found
Jul 07 02:22:10.722991 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.722922    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-crio-socket\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.722991 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.722944    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.723213 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.723008    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-crio-socket\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.723213 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:10.723043    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:10.723213 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:10.723080    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls podName:013a90cb-6fb8-4e1f-974d-04ea1d5078dc nodeName:}" failed. No retries permitted until 2026-07-07 02:22:11.223067982 +0000 UTC m=+243.866228485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkpzw" (UID: "013a90cb-6fb8-4e1f-974d-04ea1d5078dc") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:10.723213 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.723113    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-data-volume\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:10.730742 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:10.730720    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtdc\" (UniqueName: \"kubernetes.io/projected/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-api-access-kwtdc\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:11.228287 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:11.228249    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:11.228470 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:11.228314    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:11.228470 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:11.228399    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:11.228470 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:11.228408    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:22:11.228470 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:11.228465    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls podName:013a90cb-6fb8-4e1f-974d-04ea1d5078dc nodeName:}" failed. No retries permitted until 2026-07-07 02:22:12.228448829 +0000 UTC m=+244.871609329 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkpzw" (UID: "013a90cb-6fb8-4e1f-974d-04ea1d5078dc") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:11.228634 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:11.228480    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm podName:013a90cb-6fb8-4e1f-974d-04ea1d5078dc nodeName:}" failed. No retries permitted until 2026-07-07 02:22:12.228474031 +0000 UTC m=+244.871634531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkpzw" (UID: "013a90cb-6fb8-4e1f-974d-04ea1d5078dc") : configmap "kube-rbac-proxy" not found
Jul 07 02:22:12.044111 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.044082    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5n2zz"]
Jul 07 02:22:12.046170 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.046156    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.048989 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.048970    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 07 02:22:12.049587 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.049573    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 07 02:22:12.050215 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.050200    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 07 02:22:12.050215 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.050209    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 07 02:22:12.050337 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.050266    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-sb5jl\""
Jul 07 02:22:12.057513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.057492    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5n2zz"]
Jul 07 02:22:12.135030 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.134992    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdpcc\" (UniqueName: \"kubernetes.io/projected/dc3d372b-f349-4071-b005-7f84710f3e16-kube-api-access-cdpcc\") pod \"service-ca-758f599c5-5n2zz\" (UID: \"dc3d372b-f349-4071-b005-7f84710f3e16\") " pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.135030 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.135031    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc3d372b-f349-4071-b005-7f84710f3e16-signing-key\") pod \"service-ca-758f599c5-5n2zz\" (UID: \"dc3d372b-f349-4071-b005-7f84710f3e16\") " pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.135290 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.135051    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc3d372b-f349-4071-b005-7f84710f3e16-signing-cabundle\") pod \"service-ca-758f599c5-5n2zz\" (UID: \"dc3d372b-f349-4071-b005-7f84710f3e16\") " pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.236299 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.236261    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:12.236500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.236336    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:12.236500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.236364    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdpcc\" (UniqueName: \"kubernetes.io/projected/dc3d372b-f349-4071-b005-7f84710f3e16-kube-api-access-cdpcc\") pod \"service-ca-758f599c5-5n2zz\" (UID: \"dc3d372b-f349-4071-b005-7f84710f3e16\") " pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.236500 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:12.236377    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:22:12.236500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.236386    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc3d372b-f349-4071-b005-7f84710f3e16-signing-key\") pod \"service-ca-758f599c5-5n2zz\" (UID: \"dc3d372b-f349-4071-b005-7f84710f3e16\") " pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.236500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.236416    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc3d372b-f349-4071-b005-7f84710f3e16-signing-cabundle\") pod \"service-ca-758f599c5-5n2zz\" (UID: \"dc3d372b-f349-4071-b005-7f84710f3e16\") " pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.236500 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:12.236443    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm podName:013a90cb-6fb8-4e1f-974d-04ea1d5078dc nodeName:}" failed. No retries permitted until 2026-07-07 02:22:14.236423437 +0000 UTC m=+246.879583936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkpzw" (UID: "013a90cb-6fb8-4e1f-974d-04ea1d5078dc") : configmap "kube-rbac-proxy" not found
Jul 07 02:22:12.236500 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:12.236493    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:12.236793 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:12.236566    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls podName:013a90cb-6fb8-4e1f-974d-04ea1d5078dc nodeName:}" failed. No retries permitted until 2026-07-07 02:22:14.236548488 +0000 UTC m=+246.879708990 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkpzw" (UID: "013a90cb-6fb8-4e1f-974d-04ea1d5078dc") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:12.237053 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.237036    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/dc3d372b-f349-4071-b005-7f84710f3e16-signing-cabundle\") pod \"service-ca-758f599c5-5n2zz\" (UID: \"dc3d372b-f349-4071-b005-7f84710f3e16\") " pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.238894 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.238878    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/dc3d372b-f349-4071-b005-7f84710f3e16-signing-key\") pod \"service-ca-758f599c5-5n2zz\" (UID: \"dc3d372b-f349-4071-b005-7f84710f3e16\") " pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.244821 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.244800    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdpcc\" (UniqueName: \"kubernetes.io/projected/dc3d372b-f349-4071-b005-7f84710f3e16-kube-api-access-cdpcc\") pod \"service-ca-758f599c5-5n2zz\" (UID: \"dc3d372b-f349-4071-b005-7f84710f3e16\") " pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.355484 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.355401    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-5n2zz"
Jul 07 02:22:12.475067 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.475024    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-5n2zz"]
Jul 07 02:22:12.477968 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:22:12.477939    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3d372b_f349_4071_b005_7f84710f3e16.slice/crio-a0ee755758593f4c2bfa055684ed9c795e0506e77dec6683fcd8353df7ad047b WatchSource:0}: Error finding container a0ee755758593f4c2bfa055684ed9c795e0506e77dec6683fcd8353df7ad047b: Status 404 returned error can't find the container with id a0ee755758593f4c2bfa055684ed9c795e0506e77dec6683fcd8353df7ad047b
Jul 07 02:22:12.668336 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.668312    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nqvp2_e052eaa2-eca7-4894-8be6-e6b273851fa7/dns-node-resolver/0.log"
Jul 07 02:22:12.717443 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:12.717403    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5n2zz" event={"ID":"dc3d372b-f349-4071-b005-7f84710f3e16","Type":"ContainerStarted","Data":"a0ee755758593f4c2bfa055684ed9c795e0506e77dec6683fcd8353df7ad047b"}
Jul 07 02:22:13.265014 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:13.264980    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6l9zx_c703c034-4b57-4d22-a865-01ab46b75790/node-ca/0.log"
Jul 07 02:22:14.066071 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:14.066035    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5hrxw_25136846-eadf-4a17-93c9-76e0191971ff/serve-healthcheck-canary/0.log"
Jul 07 02:22:14.251551 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:14.251513    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:14.251826 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:14.251600    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:14.251826 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:14.251667    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:22:14.251826 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:14.251721    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:14.251826 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:14.251747    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm podName:013a90cb-6fb8-4e1f-974d-04ea1d5078dc nodeName:}" failed. No retries permitted until 2026-07-07 02:22:18.251726806 +0000 UTC m=+250.894887320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkpzw" (UID: "013a90cb-6fb8-4e1f-974d-04ea1d5078dc") : configmap "kube-rbac-proxy" not found
Jul 07 02:22:14.251826 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:14.251784    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls podName:013a90cb-6fb8-4e1f-974d-04ea1d5078dc nodeName:}" failed. No retries permitted until 2026-07-07 02:22:18.251765885 +0000 UTC m=+250.894926404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkpzw" (UID: "013a90cb-6fb8-4e1f-974d-04ea1d5078dc") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:14.726105 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:14.726026    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-5n2zz" event={"ID":"dc3d372b-f349-4071-b005-7f84710f3e16","Type":"ContainerStarted","Data":"6e49fb33d554ef96f681f9750b03df481138a54c56c7193b7de421303371f0f6"}
Jul 07 02:22:14.742699 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:14.742652    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-5n2zz" podStartSLOduration=0.854647057 podStartE2EDuration="2.742638529s" totalImagesPullingTime="1.887991472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:14.741861857 +0000 UTC m=+247.385022379" watchObservedRunningTime="2026-07-07 02:22:14.742638529 +0000 UTC m=+247.385799073"
Jul 07 02:22:15.710619 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:15.710523    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:15.710619 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:15.710590    2543 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:15.710994 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:15.710980    2543 scope.go:117] "RemoveContainer" containerID="bb8909aeb3d5bfbc37e0ba49189632031f235be87dabf977e93f8fbb4359e66f"
Jul 07 02:22:15.711175 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:15.711154    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-chdqk_openshift-console-operator(6c294fd4-a99a-4441-9f26-b636c3b01562)\"" pod="openshift-console-operator/console-operator-68b757fb67-chdqk" podUID="6c294fd4-a99a-4441-9f26-b636c3b01562"
Jul 07 02:22:18.283617 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:18.283588    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:18.283979 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:18.283656    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:18.283979 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:18.283705    2543 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 07 02:22:18.283979 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:18.283760    2543 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:18.283979 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:18.283764    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm podName:013a90cb-6fb8-4e1f-974d-04ea1d5078dc nodeName:}" failed. No retries permitted until 2026-07-07 02:22:26.283750193 +0000 UTC m=+258.926910692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-pkpzw" (UID: "013a90cb-6fb8-4e1f-974d-04ea1d5078dc") : configmap "kube-rbac-proxy" not found
Jul 07 02:22:18.283979 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:18.283827    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls podName:013a90cb-6fb8-4e1f-974d-04ea1d5078dc nodeName:}" failed. No retries permitted until 2026-07-07 02:22:26.283815887 +0000 UTC m=+258.926976390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-pkpzw" (UID: "013a90cb-6fb8-4e1f-974d-04ea1d5078dc") : secret "insights-runtime-extractor-tls" not found
Jul 07 02:22:19.693396 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:19.693361    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:22:19.693782 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:19.693536    2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 07 02:22:19.693782 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:19.693619    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs podName:50534db7-9f97-4088-bc8a-b33de25d0d36 nodeName:}" failed. No retries permitted until 2026-07-07 02:24:21.693599499 +0000 UTC m=+374.336760008 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs") pod "network-metrics-daemon-wllmf" (UID: "50534db7-9f97-4088-bc8a-b33de25d0d36") : secret "metrics-daemon-secret" not found
Jul 07 02:22:26.342884 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:26.342840    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:26.343495 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:26.342916    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:26.343495 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:26.343484    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:26.345459 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:26.345435    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/013a90cb-6fb8-4e1f-974d-04ea1d5078dc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pkpzw\" (UID: \"013a90cb-6fb8-4e1f-974d-04ea1d5078dc\") " pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:26.378595 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:26.378565    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pkpzw"
Jul 07 02:22:26.510162 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:26.510119    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pkpzw"]
Jul 07 02:22:26.514256 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:22:26.514223    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013a90cb_6fb8_4e1f_974d_04ea1d5078dc.slice/crio-cc877f69249ae6d48cddd94188f6e253ee14d9c71f21fd20d75ff7c2e6651f93 WatchSource:0}: Error finding container cc877f69249ae6d48cddd94188f6e253ee14d9c71f21fd20d75ff7c2e6651f93: Status 404 returned error can't find the container with id cc877f69249ae6d48cddd94188f6e253ee14d9c71f21fd20d75ff7c2e6651f93
Jul 07 02:22:26.765507 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:26.765469    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkpzw" event={"ID":"013a90cb-6fb8-4e1f-974d-04ea1d5078dc","Type":"ContainerStarted","Data":"cc877f69249ae6d48cddd94188f6e253ee14d9c71f21fd20d75ff7c2e6651f93"}
Jul 07 02:22:28.975071 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:28.974981    2543 scope.go:117] "RemoveContainer" containerID="bb8909aeb3d5bfbc37e0ba49189632031f235be87dabf977e93f8fbb4359e66f"
Jul 07 02:22:30.781632 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.781597    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkpzw" event={"ID":"013a90cb-6fb8-4e1f-974d-04ea1d5078dc","Type":"ContainerStarted","Data":"b8c5a31a1a0dccdfa7c0f52019dd9a1c8b7db24be35b020035498d00973a5115"}
Jul 07 02:22:30.783953 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.783823    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:22:30.783953 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.783887    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-chdqk" event={"ID":"6c294fd4-a99a-4441-9f26-b636c3b01562","Type":"ContainerStarted","Data":"b996f2562fa4f722e09de9fadc7be769249e086715eafd75bddcaa63a36660a7"}
Jul 07 02:22:30.784236 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.784213    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:30.789786 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.789737    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-chdqk"
Jul 07 02:22:30.844775 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.844727    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-chdqk" podStartSLOduration=23.608632351 podStartE2EDuration="25.844710725s" totalImagesPullingTime="2.236078374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:30.823419842 +0000 UTC m=+263.466580364" watchObservedRunningTime="2026-07-07 02:22:30.844710725 +0000 UTC m=+263.487871247"
Jul 07 02:22:30.975328 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.975303    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"]
Jul 07 02:22:30.978307 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.978287    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-zcxv2"]
Jul 07 02:22:30.978420 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.978407    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:30.981136 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.981117    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-zcxv2"
Jul 07 02:22:30.981559 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.981543    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 07 02:22:30.981755 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.981737    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 07 02:22:30.981925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.981910    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 07 02:22:30.982276 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.982260    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kkr4v\""
Jul 07 02:22:30.999315 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.999286    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 07 02:22:30.999426 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:30.999341    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 07 02:22:31.001108 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.001089    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-hjd8w\""
Jul 07 02:22:31.002417 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.002395    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-zcxv2"]
Jul 07 02:22:31.003513 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.003494    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"]
Jul 07 02:22:31.029604 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.029577    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 07 02:22:31.071926 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.071849    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24978d7d-4b78-4416-97ea-712c6a147898-bound-sa-token\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.071926 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.071886    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24978d7d-4b78-4416-97ea-712c6a147898-ca-trust-extracted\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.072100 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.071917    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24978d7d-4b78-4416-97ea-712c6a147898-image-registry-private-configuration\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.072100 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.071952    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24978d7d-4b78-4416-97ea-712c6a147898-registry-certificates\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.072100 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.071979    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24978d7d-4b78-4416-97ea-712c6a147898-registry-tls\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.072100 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.072060    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hdvd\" (UniqueName: \"kubernetes.io/projected/bc96cf09-5eda-4783-87a7-a2c97120b7ff-kube-api-access-2hdvd\") pod \"downloads-685bfb5f64-zcxv2\" (UID: \"bc96cf09-5eda-4783-87a7-a2c97120b7ff\") " pod="openshift-console/downloads-685bfb5f64-zcxv2"
Jul 07 02:22:31.072100 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.072099    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24978d7d-4b78-4416-97ea-712c6a147898-trusted-ca\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.072298 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.072117    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24978d7d-4b78-4416-97ea-712c6a147898-installation-pull-secrets\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.072298 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.072172    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk625\" (UniqueName: \"kubernetes.io/projected/24978d7d-4b78-4416-97ea-712c6a147898-kube-api-access-lk625\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.172502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.172469    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24978d7d-4b78-4416-97ea-712c6a147898-bound-sa-token\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.172502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.172501    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24978d7d-4b78-4416-97ea-712c6a147898-ca-trust-extracted\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.172752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.172521    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24978d7d-4b78-4416-97ea-712c6a147898-image-registry-private-configuration\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.172752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.172540    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24978d7d-4b78-4416-97ea-712c6a147898-registry-certificates\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.172752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.172569    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24978d7d-4b78-4416-97ea-712c6a147898-registry-tls\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.172752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.172597    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hdvd\" (UniqueName: \"kubernetes.io/projected/bc96cf09-5eda-4783-87a7-a2c97120b7ff-kube-api-access-2hdvd\") pod \"downloads-685bfb5f64-zcxv2\" (UID: \"bc96cf09-5eda-4783-87a7-a2c97120b7ff\") " pod="openshift-console/downloads-685bfb5f64-zcxv2"
Jul 07 02:22:31.172752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.172634    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24978d7d-4b78-4416-97ea-712c6a147898-trusted-ca\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.172752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.172656    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24978d7d-4b78-4416-97ea-712c6a147898-installation-pull-secrets\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.172752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.172688    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk625\" (UniqueName: \"kubernetes.io/projected/24978d7d-4b78-4416-97ea-712c6a147898-kube-api-access-lk625\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.173107 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.172964    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24978d7d-4b78-4416-97ea-712c6a147898-ca-trust-extracted\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.173651 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.173540    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24978d7d-4b78-4416-97ea-712c6a147898-registry-certificates\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.173651 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.173597    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24978d7d-4b78-4416-97ea-712c6a147898-trusted-ca\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.175389 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.175367    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24978d7d-4b78-4416-97ea-712c6a147898-installation-pull-secrets\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.175501 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.175393    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24978d7d-4b78-4416-97ea-712c6a147898-registry-tls\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.175501 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.175411    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24978d7d-4b78-4416-97ea-712c6a147898-image-registry-private-configuration\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.186550 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.186506    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk625\" (UniqueName: \"kubernetes.io/projected/24978d7d-4b78-4416-97ea-712c6a147898-kube-api-access-lk625\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.186669 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.186647    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hdvd\" (UniqueName: \"kubernetes.io/projected/bc96cf09-5eda-4783-87a7-a2c97120b7ff-kube-api-access-2hdvd\") pod \"downloads-685bfb5f64-zcxv2\" (UID: \"bc96cf09-5eda-4783-87a7-a2c97120b7ff\") " pod="openshift-console/downloads-685bfb5f64-zcxv2"
Jul 07 02:22:31.186840 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.186822    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24978d7d-4b78-4416-97ea-712c6a147898-bound-sa-token\") pod \"image-registry-55d6c7f4cb-fvv4h\" (UID: \"24978d7d-4b78-4416-97ea-712c6a147898\") " pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.288833 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.288809    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:31.292577 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.292560    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-zcxv2"
Jul 07 02:22:31.430102 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.430076    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"]
Jul 07 02:22:31.432671 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:22:31.432638    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24978d7d_4b78_4416_97ea_712c6a147898.slice/crio-19d6f2e9173137b7e0a4b662a7536536686b6e797fc3357d7b60e5b4ebd54b50 WatchSource:0}: Error finding container 19d6f2e9173137b7e0a4b662a7536536686b6e797fc3357d7b60e5b4ebd54b50: Status 404 returned error can't find the container with id 19d6f2e9173137b7e0a4b662a7536536686b6e797fc3357d7b60e5b4ebd54b50
Jul 07 02:22:31.443774 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.443753    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-zcxv2"]
Jul 07 02:22:31.446126 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:22:31.446105    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc96cf09_5eda_4783_87a7_a2c97120b7ff.slice/crio-31cefe992b1de1c26470b177600c1562d248b448581d5e659f9ada520dc6d8bb WatchSource:0}: Error finding container 31cefe992b1de1c26470b177600c1562d248b448581d5e659f9ada520dc6d8bb: Status 404 returned error can't find the container with id 31cefe992b1de1c26470b177600c1562d248b448581d5e659f9ada520dc6d8bb
Jul 07 02:22:31.789951 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.789906    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkpzw" event={"ID":"013a90cb-6fb8-4e1f-974d-04ea1d5078dc","Type":"ContainerStarted","Data":"c9b5a4826ed5cdbf3f9032ddacd655806be7a62d740d9b1ef55b0a453dd207a6"}
Jul 07 02:22:31.791297 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.791265    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-zcxv2" event={"ID":"bc96cf09-5eda-4783-87a7-a2c97120b7ff","Type":"ContainerStarted","Data":"31cefe992b1de1c26470b177600c1562d248b448581d5e659f9ada520dc6d8bb"}
Jul 07 02:22:31.793224 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.793178    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h" event={"ID":"24978d7d-4b78-4416-97ea-712c6a147898","Type":"ContainerStarted","Data":"beeb14f9e7b8283f8ca584d039a9b6c9013497c6d846d7baf05f500b664cd531"}
Jul 07 02:22:31.793333 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.793241    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h" event={"ID":"24978d7d-4b78-4416-97ea-712c6a147898","Type":"ContainerStarted","Data":"19d6f2e9173137b7e0a4b662a7536536686b6e797fc3357d7b60e5b4ebd54b50"}
Jul 07 02:22:31.813900 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:31.813852    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h" podStartSLOduration=1.813837503 podStartE2EDuration="1.813837503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:31.812808368 +0000 UTC m=+264.455968889" watchObservedRunningTime="2026-07-07 02:22:31.813837503 +0000 UTC m=+264.456998002"
Jul 07 02:22:32.798845 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:32.798794    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:34.809466 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:34.809423    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pkpzw" event={"ID":"013a90cb-6fb8-4e1f-974d-04ea1d5078dc","Type":"ContainerStarted","Data":"af61745c67a1c59fc231a819cdb8bf14745b1ba9fdc5ecd0f3343e5e3652d3cf"}
Jul 07 02:22:34.827293 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:34.827244    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pkpzw" podStartSLOduration=17.445098294 podStartE2EDuration="24.827228455s" totalImagesPullingTime="7.382130161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:22:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:34.826397676 +0000 UTC m=+267.469558199" watchObservedRunningTime="2026-07-07 02:22:34.827228455 +0000 UTC m=+267.470388955"
Jul 07 02:22:40.327090 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.327054    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst"]
Jul 07 02:22:40.331808 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.331789    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst"
Jul 07 02:22:40.333989 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.333967    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 07 02:22:40.334069 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.334047    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-65m7c\""
Jul 07 02:22:40.337238 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.337218    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst"]
Jul 07 02:22:40.458018 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.457974    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/15733967-d83f-416c-aae2-603f1f91cfa4-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6jbst\" (UID: \"15733967-d83f-416c-aae2-603f1f91cfa4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst"
Jul 07 02:22:40.558715 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.558677    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/15733967-d83f-416c-aae2-603f1f91cfa4-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6jbst\" (UID: \"15733967-d83f-416c-aae2-603f1f91cfa4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst"
Jul 07 02:22:40.561652 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.561628    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/15733967-d83f-416c-aae2-603f1f91cfa4-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-6jbst\" (UID: \"15733967-d83f-416c-aae2-603f1f91cfa4\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst"
Jul 07 02:22:40.644893 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.644861    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst"
Jul 07 02:22:40.801368 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.801336    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst"]
Jul 07 02:22:40.804458 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:22:40.804416    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15733967_d83f_416c_aae2_603f1f91cfa4.slice/crio-b1c60d9d7301f2b774840344d99e5e634a2af7d0c12a2c8f05bccc603280b4e4 WatchSource:0}: Error finding container b1c60d9d7301f2b774840344d99e5e634a2af7d0c12a2c8f05bccc603280b4e4: Status 404 returned error can't find the container with id b1c60d9d7301f2b774840344d99e5e634a2af7d0c12a2c8f05bccc603280b4e4
Jul 07 02:22:40.832517 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:40.832487    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst" event={"ID":"15733967-d83f-416c-aae2-603f1f91cfa4","Type":"ContainerStarted","Data":"b1c60d9d7301f2b774840344d99e5e634a2af7d0c12a2c8f05bccc603280b4e4"}
Jul 07 02:22:49.461311 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:49.461257    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dzjkl" podUID="5921583f-ac66-4059-80f0-0cf81c329d20"
Jul 07 02:22:49.867586 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:49.867551    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-zcxv2" event={"ID":"bc96cf09-5eda-4783-87a7-a2c97120b7ff","Type":"ContainerStarted","Data":"1a7360db9e259dd2dbe40930a47ce0250e8187d2ac962722dcbe3a4daa2886f9"}
Jul 07 02:22:49.867809 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:49.867780    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-zcxv2"
Jul 07 02:22:49.869560 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:49.869496    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:22:49.869560 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:49.869509    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst" event={"ID":"15733967-d83f-416c-aae2-603f1f91cfa4","Type":"ContainerStarted","Data":"370ebaa5215ad615b6e77c56abc6f02809da511b4d6250926717d3df379ac23a"}
Jul 07 02:22:49.890620 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:49.890597    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-zcxv2"
Jul 07 02:22:49.905735 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:49.905680    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-zcxv2" podStartSLOduration=2.438795308 podStartE2EDuration="19.9056595s" totalImagesPullingTime="17.466864192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:49.887684719 +0000 UTC m=+282.530845242" watchObservedRunningTime="2026-07-07 02:22:49.9056595 +0000 UTC m=+282.548820022"
Jul 07 02:22:49.905893 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:49.905798    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst" podStartSLOduration=1.844788959 podStartE2EDuration="9.905786672s" totalImagesPullingTime="8.060997713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:49.904101982 +0000 UTC m=+282.547262513" watchObservedRunningTime="2026-07-07 02:22:49.905786672 +0000 UTC m=+282.548947194"
Jul 07 02:22:50.874269 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:50.874232    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst"
Jul 07 02:22:50.878275 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:50.878248    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-6jbst"
Jul 07 02:22:51.296118 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:51.296037    2543 patch_prober.go:28] interesting pod/image-registry-55d6c7f4cb-fvv4h container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 07 02:22:51.296118 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:51.296096    2543 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h" podUID="24978d7d-4b78-4416-97ea-712c6a147898" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 07 02:22:52.665388 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:52.665342    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:22:52.667963 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:52.667936    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5921583f-ac66-4059-80f0-0cf81c329d20-metrics-tls\") pod \"dns-default-dzjkl\" (UID: \"5921583f-ac66-4059-80f0-0cf81c329d20\") " pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:22:52.874547 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:52.874517    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l8w8v\""
Jul 07 02:22:52.882313 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:52.882286    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:22:53.067990 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:53.067954    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dzjkl"]
Jul 07 02:22:53.071636 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:22:53.071611    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5921583f_ac66_4059_80f0_0cf81c329d20.slice/crio-57c5b0e50802e9e3aae37734e07e2dd295baa78e0278f052de25ac1599e2d64b WatchSource:0}: Error finding container 57c5b0e50802e9e3aae37734e07e2dd295baa78e0278f052de25ac1599e2d64b: Status 404 returned error can't find the container with id 57c5b0e50802e9e3aae37734e07e2dd295baa78e0278f052de25ac1599e2d64b
Jul 07 02:22:53.811128 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:53.810878    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55d6c7f4cb-fvv4h"
Jul 07 02:22:53.887793 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:53.887656    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dzjkl" event={"ID":"5921583f-ac66-4059-80f0-0cf81c329d20","Type":"ContainerStarted","Data":"57c5b0e50802e9e3aae37734e07e2dd295baa78e0278f052de25ac1599e2d64b"}
Jul 07 02:22:56.769634 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.769592    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"]
Jul 07 02:22:56.804954 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.804916    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"]
Jul 07 02:22:56.805107 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.804970    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mc25k"]
Jul 07 02:22:56.806418 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.806393    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:56.809413 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.809392    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-t5bzj\""
Jul 07 02:22:56.809623 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.809607    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 07 02:22:56.809883 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.809858    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 07 02:22:56.811120 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.810522    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 07 02:22:56.811120 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.810734    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 07 02:22:56.811120 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.810970    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 07 02:22:56.826228 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.826203    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:56.829304 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.829281    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 07 02:22:56.829583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.829561    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fpv7g\""
Jul 07 02:22:56.829750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.829286    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 07 02:22:56.829989 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.829953    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 07 02:22:56.901725 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.901681    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dzjkl" event={"ID":"5921583f-ac66-4059-80f0-0cf81c329d20","Type":"ContainerStarted","Data":"23bd82e9c2262b1f6c0c0bd308ae48f650906285cac97c107b232fe9744872c1"}
Jul 07 02:22:56.901917 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.901751    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dzjkl" event={"ID":"5921583f-ac66-4059-80f0-0cf81c329d20","Type":"ContainerStarted","Data":"5262f62bf6bf71b00eed05cd296038bc3d72c94cd4bf73dff8df7d2867a0c039"}
Jul 07 02:22:56.902144 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.902122    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:22:56.904753 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.904722    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-tls\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:56.904859 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.904771    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc39428c-c98a-46b6-ad0d-ba35deb34614-metrics-client-ca\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:56.904859 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.904848    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc39428c-c98a-46b6-ad0d-ba35deb34614-sys\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:56.904971 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.904891    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-accelerators-collector-config\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:56.904971 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.904920    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wc8\" (UniqueName: \"kubernetes.io/projected/cc39428c-c98a-46b6-ad0d-ba35deb34614-kube-api-access-89wc8\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:56.905067 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.904984    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-textfile\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:56.905067 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.905035    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:56.905172 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.905083    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:56.905172 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.905114    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc39428c-c98a-46b6-ad0d-ba35deb34614-root\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:56.905172 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.905160    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:56.905366 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.905246    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-wtmp\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:56.905366 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.905285    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:56.905366 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.905312    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv8h2\" (UniqueName: \"kubernetes.io/projected/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-kube-api-access-hv8h2\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:56.919641 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:56.919586    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dzjkl" podStartSLOduration=252.234172778 podStartE2EDuration="4m14.919568784s" totalImagesPullingTime="2.685396006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:56.918594907 +0000 UTC m=+289.561755433" watchObservedRunningTime="2026-07-07 02:22:56.919568784 +0000 UTC m=+289.562729301"
Jul 07 02:22:57.005839 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.005797    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-wtmp\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.006021 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.005859    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:57.006021 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.005889    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv8h2\" (UniqueName: \"kubernetes.io/projected/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-kube-api-access-hv8h2\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:57.006021 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.005958    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-tls\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.006021 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006002    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc39428c-c98a-46b6-ad0d-ba35deb34614-metrics-client-ca\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.006260 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006035    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc39428c-c98a-46b6-ad0d-ba35deb34614-sys\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.006260 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006072    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-accelerators-collector-config\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.006260 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006098    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wc8\" (UniqueName: \"kubernetes.io/projected/cc39428c-c98a-46b6-ad0d-ba35deb34614-kube-api-access-89wc8\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.006260 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006141    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-textfile\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.006260 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006170    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:57.006260 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006227    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.006260 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006259    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc39428c-c98a-46b6-ad0d-ba35deb34614-root\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.006576 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006286    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:57.008658 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006833    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cc39428c-c98a-46b6-ad0d-ba35deb34614-sys\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.008658 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.006963    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-wtmp\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.008658 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:57.007056    2543 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 07 02:22:57.008658 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:57.007111    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-openshift-state-metrics-tls podName:cf095b1e-ae19-46b7-a5bc-24828d33d3e9 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:57.507092406 +0000 UTC m=+290.150252917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-cbxjp" (UID: "cf095b1e-ae19-46b7-a5bc-24828d33d3e9") : secret "openshift-state-metrics-tls" not found
Jul 07 02:22:57.008658 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.008001    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc39428c-c98a-46b6-ad0d-ba35deb34614-metrics-client-ca\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.008658 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.008147    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-textfile\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.008658 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:57.008312    2543 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 07 02:22:57.008658 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:22:57.008360    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-tls podName:cc39428c-c98a-46b6-ad0d-ba35deb34614 nodeName:}" failed. No retries permitted until 2026-07-07 02:22:57.508344577 +0000 UTC m=+290.151505078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-tls") pod "node-exporter-mc25k" (UID: "cc39428c-c98a-46b6-ad0d-ba35deb34614") : secret "node-exporter-tls" not found
Jul 07 02:22:57.008658 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.008646    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-accelerators-collector-config\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.009372 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.009329    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/cc39428c-c98a-46b6-ad0d-ba35deb34614-root\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.009865 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.009838    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:57.010754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.010704    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:57.012672 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.012642    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.027904 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.027839    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv8h2\" (UniqueName: \"kubernetes.io/projected/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-kube-api-access-hv8h2\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:57.027904 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.027886    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wc8\" (UniqueName: \"kubernetes.io/projected/cc39428c-c98a-46b6-ad0d-ba35deb34614-kube-api-access-89wc8\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.511419 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.511381    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:57.511730 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.511455    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-tls\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.514440 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.514413    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/cc39428c-c98a-46b6-ad0d-ba35deb34614-node-exporter-tls\") pod \"node-exporter-mc25k\" (UID: \"cc39428c-c98a-46b6-ad0d-ba35deb34614\") " pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.514568 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.514507    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/cf095b1e-ae19-46b7-a5bc-24828d33d3e9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-cbxjp\" (UID: \"cf095b1e-ae19-46b7-a5bc-24828d33d3e9\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:57.719986 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.719948    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"
Jul 07 02:22:57.740385 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.740329    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mc25k"
Jul 07 02:22:57.777272 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:22:57.777237    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc39428c_c98a_46b6_ad0d_ba35deb34614.slice/crio-dc114eb91e50b8d532384defe7cd908abe6f8c10af0681e578cf8b2138eacbfd WatchSource:0}: Error finding container dc114eb91e50b8d532384defe7cd908abe6f8c10af0681e578cf8b2138eacbfd: Status 404 returned error can't find the container with id dc114eb91e50b8d532384defe7cd908abe6f8c10af0681e578cf8b2138eacbfd
Jul 07 02:22:57.908101 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.908054    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc25k" event={"ID":"cc39428c-c98a-46b6-ad0d-ba35deb34614","Type":"ContainerStarted","Data":"dc114eb91e50b8d532384defe7cd908abe6f8c10af0681e578cf8b2138eacbfd"}
Jul 07 02:22:57.917721 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:57.917620    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp"]
Jul 07 02:22:57.921109 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:22:57.921080    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf095b1e_ae19_46b7_a5bc_24828d33d3e9.slice/crio-70479fbeabcec27930ab019d9b980cee6d2021d275b585f611f13098cda8fee3 WatchSource:0}: Error finding container 70479fbeabcec27930ab019d9b980cee6d2021d275b585f611f13098cda8fee3: Status 404 returned error can't find the container with id 70479fbeabcec27930ab019d9b980cee6d2021d275b585f611f13098cda8fee3
Jul 07 02:22:58.914254 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:58.914216    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp" event={"ID":"cf095b1e-ae19-46b7-a5bc-24828d33d3e9","Type":"ContainerStarted","Data":"3c915225e2f4bd56ad9c3448f798034560fe8d03a317f7dd73b8b48b6f09cbe9"}
Jul 07 02:22:58.914682 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:58.914286    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp" event={"ID":"cf095b1e-ae19-46b7-a5bc-24828d33d3e9","Type":"ContainerStarted","Data":"caa3ba6e501a5e794ef152de8e09c607aa76e92d7f44ec6c1f9950dcd7f7a562"}
Jul 07 02:22:58.914682 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:58.914309    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp" event={"ID":"cf095b1e-ae19-46b7-a5bc-24828d33d3e9","Type":"ContainerStarted","Data":"70479fbeabcec27930ab019d9b980cee6d2021d275b585f611f13098cda8fee3"}
Jul 07 02:22:59.924612 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:59.924578    2543 generic.go:356] "Generic (PLEG): container finished" podID="cc39428c-c98a-46b6-ad0d-ba35deb34614" containerID="3eb98f75997caed5419b1d5ce5cacf9e6e63c5b8269af72086692b1b92d9fb18" exitCode=0
Jul 07 02:22:59.925114 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:59.924667    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc25k" event={"ID":"cc39428c-c98a-46b6-ad0d-ba35deb34614","Type":"ContainerDied","Data":"3eb98f75997caed5419b1d5ce5cacf9e6e63c5b8269af72086692b1b92d9fb18"}
Jul 07 02:22:59.927486 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:59.927430    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp" event={"ID":"cf095b1e-ae19-46b7-a5bc-24828d33d3e9","Type":"ContainerStarted","Data":"8486094a7f726c64e4b15e191bfef295f0d3692f301aabb8b0f1a25c7ad2eea1"}
Jul 07 02:22:59.963432 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:22:59.963221    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-cbxjp" podStartSLOduration=2.358723415 podStartE2EDuration="3.963179509s" totalImagesPullingTime="1.604456094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:22:59.961678623 +0000 UTC m=+292.604839150" watchObservedRunningTime="2026-07-07 02:22:59.963179509 +0000 UTC m=+292.606340031"
Jul 07 02:23:00.933036 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:23:00.933002    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc25k" event={"ID":"cc39428c-c98a-46b6-ad0d-ba35deb34614","Type":"ContainerStarted","Data":"60ab01297425dd2cd09cfcf6392b22ca8d9ec098d2d66418fd0ac7d0f80c0dd3"}
Jul 07 02:23:00.933417 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:23:00.933060    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mc25k" event={"ID":"cc39428c-c98a-46b6-ad0d-ba35deb34614","Type":"ContainerStarted","Data":"f3cdf29966c11c1d3d1682bf6672caf9a6cd8ed2ce3c769d9088845001abbc6b"}
Jul 07 02:23:00.960587 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:23:00.960532    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mc25k" podStartSLOduration=3.152750585 podStartE2EDuration="4.960516894s" totalImagesPullingTime="1.709361946s" totalInitContainerRuntime="98.404363ms" isStatefulPod=true podCreationTimestamp="2026-07-07 02:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:23:00.958733337 +0000 UTC m=+293.601893861" watchObservedRunningTime="2026-07-07 02:23:00.960516894 +0000 UTC m=+293.603677453"
Jul 07 02:23:04.975489 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:23:04.975440    2543 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-wllmf" podUID="50534db7-9f97-4088-bc8a-b33de25d0d36"
Jul 07 02:23:06.910833 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:23:06.910808    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dzjkl"
Jul 07 02:23:07.804496 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:23:07.804466    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:23:07.805413 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:23:07.805388    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:23:07.815077 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:23:07.815052    2543 kubelet.go:1648] "Image garbage collection succeeded"
Jul 07 02:23:15.975104 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:23:15.975068    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:24:21.123724 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.123684    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-567b4cc688-4j2sv"]
Jul 07 02:24:21.126143 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.126124    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.128627 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.128604    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 07 02:24:21.128783 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.128656    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tr9lr\""
Jul 07 02:24:21.128889 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.128674    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 07 02:24:21.128889 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.128686    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 07 02:24:21.129005 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.128724    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 07 02:24:21.129005 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.128768    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 07 02:24:21.132754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.132730    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 07 02:24:21.140020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.139954    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-567b4cc688-4j2sv"]
Jul 07 02:24:21.184743 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.184707    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-federate-client-tls\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.184743 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.184745    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-telemeter-client-tls\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.184989 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.184761    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.184989 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.184815    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-metrics-client-ca\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.184989 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.184858    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-secret-telemeter-client\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.184989 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.184923    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-serving-certs-ca-bundle\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.184989 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.184944    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c268f\" (UniqueName: \"kubernetes.io/projected/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-kube-api-access-c268f\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.184989 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.184969    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.285511 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.285463    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-secret-telemeter-client\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.285715 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.285532    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-serving-certs-ca-bundle\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.285715 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.285552    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c268f\" (UniqueName: \"kubernetes.io/projected/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-kube-api-access-c268f\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.285715 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.285581    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.285715 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.285621    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-federate-client-tls\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.285715 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.285650    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-telemeter-client-tls\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.285715 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.285670    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.285715 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.285697    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-metrics-client-ca\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.286409 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.286376    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-metrics-client-ca\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.286545 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.286523    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-serving-certs-ca-bundle\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.286919 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.286896    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-telemeter-trusted-ca-bundle\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.288321 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.288299    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-secret-telemeter-client\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.288415 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.288400    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-telemeter-client-tls\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.288624 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.288607    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.288668 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.288649    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-federate-client-tls\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.294142 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.294121    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c268f\" (UniqueName: \"kubernetes.io/projected/b63caec1-c0c2-4f6c-b210-f1940d7cfff3-kube-api-access-c268f\") pod \"telemeter-client-567b4cc688-4j2sv\" (UID: \"b63caec1-c0c2-4f6c-b210-f1940d7cfff3\") " pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.436389 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.436297    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv"
Jul 07 02:24:21.577437 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.577407    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-567b4cc688-4j2sv"]
Jul 07 02:24:21.583031 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.583011    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:24:21.788889 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.788800    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:24:21.791227 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.791202    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50534db7-9f97-4088-bc8a-b33de25d0d36-metrics-certs\") pod \"network-metrics-daemon-wllmf\" (UID: \"50534db7-9f97-4088-bc8a-b33de25d0d36\") " pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:24:21.978388 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.978361    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dwgqf\""
Jul 07 02:24:21.986550 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:21.986521    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wllmf"
Jul 07 02:24:22.114765 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:22.114730    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wllmf"]
Jul 07 02:24:22.117465 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:24:22.117436    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50534db7_9f97_4088_bc8a_b33de25d0d36.slice/crio-bf943efd7d16c51716860891e965342acaf7784263cd765000fd787dec6eb757 WatchSource:0}: Error finding container bf943efd7d16c51716860891e965342acaf7784263cd765000fd787dec6eb757: Status 404 returned error can't find the container with id bf943efd7d16c51716860891e965342acaf7784263cd765000fd787dec6eb757
Jul 07 02:24:22.214677 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:22.214643    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wllmf" event={"ID":"50534db7-9f97-4088-bc8a-b33de25d0d36","Type":"ContainerStarted","Data":"bf943efd7d16c51716860891e965342acaf7784263cd765000fd787dec6eb757"}
Jul 07 02:24:22.215807 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:22.215784    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv" event={"ID":"b63caec1-c0c2-4f6c-b210-f1940d7cfff3","Type":"ContainerStarted","Data":"cc3510eb1e2347c53de935bd77c78fbb45ab70b2c436522ea13060bbce07fc90"}
Jul 07 02:24:24.225270 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:24.225229    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv" event={"ID":"b63caec1-c0c2-4f6c-b210-f1940d7cfff3","Type":"ContainerStarted","Data":"fd3d48915862b0118867e805ec465b63c1a813409befe7f3aac841b51f2a45c7"}
Jul 07 02:24:24.227076 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:24.227032    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wllmf" event={"ID":"50534db7-9f97-4088-bc8a-b33de25d0d36","Type":"ContainerStarted","Data":"1af1b2a08d0c253ecab72e4bce847caed1c3d0ae12a4c53ad4a4474083465918"}
Jul 07 02:24:24.227207 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:24.227093    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wllmf" event={"ID":"50534db7-9f97-4088-bc8a-b33de25d0d36","Type":"ContainerStarted","Data":"233858bcb8be22875bc34f2a90563e98cc0abb6ceeea28b6332663b2e681f06a"}
Jul 07 02:24:24.241854 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:24.241805    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wllmf" podStartSLOduration=376.179907558 podStartE2EDuration="6m17.241791363s" totalImagesPullingTime="1.061883805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:18:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:24.241020273 +0000 UTC m=+376.884180810" watchObservedRunningTime="2026-07-07 02:24:24.241791363 +0000 UTC m=+376.884951885"
Jul 07 02:24:26.237026 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:26.236984    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv" event={"ID":"b63caec1-c0c2-4f6c-b210-f1940d7cfff3","Type":"ContainerStarted","Data":"b76c381a84bccfdfdd760834e7aec081d4f12419cf2f1f5be3de81a224b45a0c"}
Jul 07 02:24:26.237560 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:26.237051    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv" event={"ID":"b63caec1-c0c2-4f6c-b210-f1940d7cfff3","Type":"ContainerStarted","Data":"4c1be6eddd673b6da02f0f9dd42735bce5f473069306ee0e49c2dc5f68e27fde"}
Jul 07 02:24:26.259987 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:24:26.259931    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-567b4cc688-4j2sv" podStartSLOduration=1.43399824 podStartE2EDuration="5.25991582s" totalImagesPullingTime="3.82591758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:24:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:24:26.258983994 +0000 UTC m=+378.902144545" watchObservedRunningTime="2026-07-07 02:24:26.25991582 +0000 UTC m=+378.903076344"
Jul 07 02:27:09.242098 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.242015    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"]
Jul 07 02:27:09.244560 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.244531    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:09.248596 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.248574    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 07 02:27:09.249300 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.249286    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 07 02:27:09.249420 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.249402    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 07 02:27:09.249468 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.249441    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 07 02:27:09.249522 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.249510    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-vwjlh\""
Jul 07 02:27:09.250020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.250006    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 07 02:27:09.262607 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.262580    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"]
Jul 07 02:27:09.398640 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.398601    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-mk7xc\" (UID: \"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:09.398825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.398655    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-certificates\") pod \"keda-metrics-apiserver-559f75f947-mk7xc\" (UID: \"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:09.398825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.398726    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klcsj\" (UniqueName: \"kubernetes.io/projected/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-kube-api-access-klcsj\") pod \"keda-metrics-apiserver-559f75f947-mk7xc\" (UID: \"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:09.499480 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.499398    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klcsj\" (UniqueName: \"kubernetes.io/projected/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-kube-api-access-klcsj\") pod \"keda-metrics-apiserver-559f75f947-mk7xc\" (UID: \"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:09.499480 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.499450    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-mk7xc\" (UID: \"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:09.499669 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.499488    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-certificates\") pod \"keda-metrics-apiserver-559f75f947-mk7xc\" (UID: \"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:09.499669 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:27:09.499577    2543 secret.go:281] references non-existent secret key: tls.crt
Jul 07 02:27:09.499669 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:27:09.499590    2543 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 07 02:27:09.499669 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:27:09.499605    2543 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc: references non-existent secret key: tls.crt
Jul 07 02:27:09.499669 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:27:09.499655    2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-certificates podName:947a6eca-5c2d-4c47-9160-c4ffc80d2ea9 nodeName:}" failed. No retries permitted until 2026-07-07 02:27:09.999640423 +0000 UTC m=+542.642800923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-certificates") pod "keda-metrics-apiserver-559f75f947-mk7xc" (UID: "947a6eca-5c2d-4c47-9160-c4ffc80d2ea9") : references non-existent secret key: tls.crt
Jul 07 02:27:09.499886 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.499867    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-temp-vol\") pod \"keda-metrics-apiserver-559f75f947-mk7xc\" (UID: \"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:09.514503 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:09.514463    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klcsj\" (UniqueName: \"kubernetes.io/projected/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-kube-api-access-klcsj\") pod \"keda-metrics-apiserver-559f75f947-mk7xc\" (UID: \"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:10.004449 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:10.004408    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-certificates\") pod \"keda-metrics-apiserver-559f75f947-mk7xc\" (UID: \"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:10.007000 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:10.006973    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/947a6eca-5c2d-4c47-9160-c4ffc80d2ea9-certificates\") pod \"keda-metrics-apiserver-559f75f947-mk7xc\" (UID: \"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9\") " pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:10.154381 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:10.154330    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:10.315203 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:10.313280    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"]
Jul 07 02:27:10.319699 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:27:10.319636    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod947a6eca_5c2d_4c47_9160_c4ffc80d2ea9.slice/crio-dd072139793c50b8e746b3f05eb3df03c03a672111f7e1546ef7faacc29351d5 WatchSource:0}: Error finding container dd072139793c50b8e746b3f05eb3df03c03a672111f7e1546ef7faacc29351d5: Status 404 returned error can't find the container with id dd072139793c50b8e746b3f05eb3df03c03a672111f7e1546ef7faacc29351d5
Jul 07 02:27:10.832518 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:10.832479    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc" event={"ID":"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9","Type":"ContainerStarted","Data":"dd072139793c50b8e746b3f05eb3df03c03a672111f7e1546ef7faacc29351d5"}
Jul 07 02:27:13.846253 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:13.846150    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc" event={"ID":"947a6eca-5c2d-4c47-9160-c4ffc80d2ea9","Type":"ContainerStarted","Data":"bccc9cf9a7672059d4cb67f12c7b5ebe9c098d6d79f92acf74734ad1132de138"}
Jul 07 02:27:13.846612 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:13.846311    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:27:13.872894 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:13.872847    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc" podStartSLOduration=1.768880535 podStartE2EDuration="4.872831616s" totalImagesPullingTime="3.103951081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:27:13.872638927 +0000 UTC m=+546.515799448" watchObservedRunningTime="2026-07-07 02:27:13.872831616 +0000 UTC m=+546.515992138"
Jul 07 02:27:24.854726 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:27:24.854685    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-559f75f947-mk7xc"
Jul 07 02:28:07.830452 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:07.830416    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:28:07.830994 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:07.830495    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:28:20.581581 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.581542    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-2xf52"]
Jul 07 02:28:20.590645 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.590622    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:28:20.593593 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.593572    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 07 02:28:20.594358 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.594340    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 07 02:28:20.594451 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.594349    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 07 02:28:20.594598 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.594581    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-84kkc\""
Jul 07 02:28:20.606552 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.606526    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-2xf52"]
Jul 07 02:28:20.669952 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.669919    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-m5wwv"]
Jul 07 02:28:20.672760 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.672741    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg4tw\" (UniqueName: \"kubernetes.io/projected/102da792-ed38-4de2-ad95-8c2eef04d1b5-kube-api-access-jg4tw\") pod \"kserve-controller-manager-f8f7445f4-2xf52\" (UID: \"102da792-ed38-4de2-ad95-8c2eef04d1b5\") " pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:28:20.672827 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.672788    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/102da792-ed38-4de2-ad95-8c2eef04d1b5-cert\") pod \"kserve-controller-manager-f8f7445f4-2xf52\" (UID: \"102da792-ed38-4de2-ad95-8c2eef04d1b5\") " pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:28:20.678651 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.678635    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-m5wwv"
Jul 07 02:28:20.683981 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.683962    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-zl4w7\""
Jul 07 02:28:20.691770 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.691746    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 07 02:28:20.692821 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.692800    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-m5wwv"]
Jul 07 02:28:20.773461 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.773427    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4tw\" (UniqueName: \"kubernetes.io/projected/102da792-ed38-4de2-ad95-8c2eef04d1b5-kube-api-access-jg4tw\") pod \"kserve-controller-manager-f8f7445f4-2xf52\" (UID: \"102da792-ed38-4de2-ad95-8c2eef04d1b5\") " pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:28:20.773637 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.773478    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdz4\" (UniqueName: \"kubernetes.io/projected/be4a725e-d246-44d1-ad23-8c7648e7ae7a-kube-api-access-cfdz4\") pod \"seaweedfs-64568bcd49-m5wwv\" (UID: \"be4a725e-d246-44d1-ad23-8c7648e7ae7a\") " pod="kserve/seaweedfs-64568bcd49-m5wwv"
Jul 07 02:28:20.773637 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.773522    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/be4a725e-d246-44d1-ad23-8c7648e7ae7a-data\") pod \"seaweedfs-64568bcd49-m5wwv\" (UID: \"be4a725e-d246-44d1-ad23-8c7648e7ae7a\") " pod="kserve/seaweedfs-64568bcd49-m5wwv"
Jul 07 02:28:20.773637 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.773631    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/102da792-ed38-4de2-ad95-8c2eef04d1b5-cert\") pod \"kserve-controller-manager-f8f7445f4-2xf52\" (UID: \"102da792-ed38-4de2-ad95-8c2eef04d1b5\") " pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:28:20.776106 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.776078    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/102da792-ed38-4de2-ad95-8c2eef04d1b5-cert\") pod \"kserve-controller-manager-f8f7445f4-2xf52\" (UID: \"102da792-ed38-4de2-ad95-8c2eef04d1b5\") " pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:28:20.784574 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.784551    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg4tw\" (UniqueName: \"kubernetes.io/projected/102da792-ed38-4de2-ad95-8c2eef04d1b5-kube-api-access-jg4tw\") pod \"kserve-controller-manager-f8f7445f4-2xf52\" (UID: \"102da792-ed38-4de2-ad95-8c2eef04d1b5\") " pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:28:20.874034 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.873987    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdz4\" (UniqueName: \"kubernetes.io/projected/be4a725e-d246-44d1-ad23-8c7648e7ae7a-kube-api-access-cfdz4\") pod \"seaweedfs-64568bcd49-m5wwv\" (UID: \"be4a725e-d246-44d1-ad23-8c7648e7ae7a\") " pod="kserve/seaweedfs-64568bcd49-m5wwv"
Jul 07 02:28:20.874243 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.874049    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/be4a725e-d246-44d1-ad23-8c7648e7ae7a-data\") pod \"seaweedfs-64568bcd49-m5wwv\" (UID: \"be4a725e-d246-44d1-ad23-8c7648e7ae7a\") " pod="kserve/seaweedfs-64568bcd49-m5wwv"
Jul 07 02:28:20.874510 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.874487    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/be4a725e-d246-44d1-ad23-8c7648e7ae7a-data\") pod \"seaweedfs-64568bcd49-m5wwv\" (UID: \"be4a725e-d246-44d1-ad23-8c7648e7ae7a\") " pod="kserve/seaweedfs-64568bcd49-m5wwv"
Jul 07 02:28:20.885346 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.885322    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdz4\" (UniqueName: \"kubernetes.io/projected/be4a725e-d246-44d1-ad23-8c7648e7ae7a-kube-api-access-cfdz4\") pod \"seaweedfs-64568bcd49-m5wwv\" (UID: \"be4a725e-d246-44d1-ad23-8c7648e7ae7a\") " pod="kserve/seaweedfs-64568bcd49-m5wwv"
Jul 07 02:28:20.901162 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.901130    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:28:20.988223 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:20.987400    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-m5wwv"
Jul 07 02:28:21.059689 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:21.059625    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-2xf52"]
Jul 07 02:28:21.062673 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:28:21.062633    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102da792_ed38_4de2_ad95_8c2eef04d1b5.slice/crio-97b6915e5ad25200e4662422a51f6715ad8c610c571afd82c8d5262ba5cbc044 WatchSource:0}: Error finding container 97b6915e5ad25200e4662422a51f6715ad8c610c571afd82c8d5262ba5cbc044: Status 404 returned error can't find the container with id 97b6915e5ad25200e4662422a51f6715ad8c610c571afd82c8d5262ba5cbc044
Jul 07 02:28:21.096877 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:21.096835    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52" event={"ID":"102da792-ed38-4de2-ad95-8c2eef04d1b5","Type":"ContainerStarted","Data":"97b6915e5ad25200e4662422a51f6715ad8c610c571afd82c8d5262ba5cbc044"}
Jul 07 02:28:21.141226 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:21.141112    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-m5wwv"]
Jul 07 02:28:21.144805 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:28:21.144777    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4a725e_d246_44d1_ad23_8c7648e7ae7a.slice/crio-f293506cfddc7a84609c13c2f2e1a14c76fdc3bb6c2754da0f88a647d38c4ee8 WatchSource:0}: Error finding container f293506cfddc7a84609c13c2f2e1a14c76fdc3bb6c2754da0f88a647d38c4ee8: Status 404 returned error can't find the container with id f293506cfddc7a84609c13c2f2e1a14c76fdc3bb6c2754da0f88a647d38c4ee8
Jul 07 02:28:22.104047 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:22.104009    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-m5wwv" event={"ID":"be4a725e-d246-44d1-ad23-8c7648e7ae7a","Type":"ContainerStarted","Data":"f293506cfddc7a84609c13c2f2e1a14c76fdc3bb6c2754da0f88a647d38c4ee8"}
Jul 07 02:28:25.116911 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:25.116876    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-m5wwv" event={"ID":"be4a725e-d246-44d1-ad23-8c7648e7ae7a","Type":"ContainerStarted","Data":"55a23ef47a93a5d0f517c2ac81003be59c13cc7262e7f7bb8da7724ddbf39d8a"}
Jul 07 02:28:25.117368 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:25.117023    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-m5wwv"
Jul 07 02:28:25.118406 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:25.118383    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52" event={"ID":"102da792-ed38-4de2-ad95-8c2eef04d1b5","Type":"ContainerStarted","Data":"ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363"}
Jul 07 02:28:25.118489 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:25.118448    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:28:25.138846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:25.138799    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-m5wwv" podStartSLOduration=1.5129346080000001 podStartE2EDuration="5.138784307s" totalImagesPullingTime="3.625849699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:28:25.13801515 +0000 UTC m=+617.781175674" watchObservedRunningTime="2026-07-07 02:28:25.138784307 +0000 UTC m=+617.781944829"
Jul 07 02:28:25.159047 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:25.159000    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52" podStartSLOduration=1.538646202 podStartE2EDuration="5.158984285s" totalImagesPullingTime="3.620338083s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:28:25.157836806 +0000 UTC m=+617.800997328" watchObservedRunningTime="2026-07-07 02:28:25.158984285 +0000 UTC m=+617.802144856"
Jul 07 02:28:31.124729 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:31.124651    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-m5wwv"
Jul 07 02:28:56.128047 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:28:56.128018    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:29:10.859201 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:10.859164    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"]
Jul 07 02:29:10.861274 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:10.861256    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"
Jul 07 02:29:10.864335 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:10.864314    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:29:10.864561 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:10.864545    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-k9zpf\""
Jul 07 02:29:10.864611 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:10.864581    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 07 02:29:10.878298 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:10.878275    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"]
Jul 07 02:29:10.964649 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:10.964605    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prskj\" (UniqueName: \"kubernetes.io/projected/0b8bc649-11a8-490b-a804-485147d7527d-kube-api-access-prskj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-kmmnz\" (UID: \"0b8bc649-11a8-490b-a804-485147d7527d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"
Jul 07 02:29:10.964832 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:10.964660    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b8bc649-11a8-490b-a804-485147d7527d-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-kmmnz\" (UID: \"0b8bc649-11a8-490b-a804-485147d7527d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"
Jul 07 02:29:11.065895 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:11.065860    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b8bc649-11a8-490b-a804-485147d7527d-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-kmmnz\" (UID: \"0b8bc649-11a8-490b-a804-485147d7527d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"
Jul 07 02:29:11.066079 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:11.065962    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prskj\" (UniqueName: \"kubernetes.io/projected/0b8bc649-11a8-490b-a804-485147d7527d-kube-api-access-prskj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-kmmnz\" (UID: \"0b8bc649-11a8-490b-a804-485147d7527d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"
Jul 07 02:29:11.066355 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:11.066333    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b8bc649-11a8-490b-a804-485147d7527d-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-kmmnz\" (UID: \"0b8bc649-11a8-490b-a804-485147d7527d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"
Jul 07 02:29:11.078635 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:11.078614    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prskj\" (UniqueName: \"kubernetes.io/projected/0b8bc649-11a8-490b-a804-485147d7527d-kube-api-access-prskj\") pod \"cert-manager-operator-controller-manager-684f79c9b9-kmmnz\" (UID: \"0b8bc649-11a8-490b-a804-485147d7527d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"
Jul 07 02:29:11.171151 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:11.171110    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"
Jul 07 02:29:11.308127 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:11.308104    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz"]
Jul 07 02:29:11.310767 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:29:11.310737    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b8bc649_11a8_490b_a804_485147d7527d.slice/crio-91e494aba66f4c73da45aee0f232c79f3efaebad9b7664e99ef08c59fb9d6258 WatchSource:0}: Error finding container 91e494aba66f4c73da45aee0f232c79f3efaebad9b7664e99ef08c59fb9d6258: Status 404 returned error can't find the container with id 91e494aba66f4c73da45aee0f232c79f3efaebad9b7664e99ef08c59fb9d6258
Jul 07 02:29:12.296766 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:12.296727    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz" event={"ID":"0b8bc649-11a8-490b-a804-485147d7527d","Type":"ContainerStarted","Data":"91e494aba66f4c73da45aee0f232c79f3efaebad9b7664e99ef08c59fb9d6258"}
Jul 07 02:29:13.309162 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:13.309127    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz" event={"ID":"0b8bc649-11a8-490b-a804-485147d7527d","Type":"ContainerStarted","Data":"b3c6b8ddcff7575acd49a76cdac955e56974a50dfe7408731b4d6ef61abcbf84"}
Jul 07 02:29:13.336799 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:13.336730    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-kmmnz" podStartSLOduration=1.557965306 podStartE2EDuration="3.336707549s" totalImagesPullingTime="1.778742243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:13.334272455 +0000 UTC m=+665.977432978" watchObservedRunningTime="2026-07-07 02:29:13.336707549 +0000 UTC m=+665.979868074"
Jul 07 02:29:17.441531 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.441502    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"]
Jul 07 02:29:17.443794 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.443777    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"
Jul 07 02:29:17.446379 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.446354    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 07 02:29:17.446518 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.446406    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 07 02:29:17.447461 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.447442    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-wtrtw\""
Jul 07 02:29:17.455967 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.455945    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"]
Jul 07 02:29:17.523998 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.523962    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac0f3001-a96e-41ca-bcf7-310e07554447-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wb4f9\" (UID: \"ac0f3001-a96e-41ca-bcf7-310e07554447\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"
Jul 07 02:29:17.524227 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.524077    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws56n\" (UniqueName: \"kubernetes.io/projected/ac0f3001-a96e-41ca-bcf7-310e07554447-kube-api-access-ws56n\") pod \"cert-manager-cainjector-559cdd8c87-wb4f9\" (UID: \"ac0f3001-a96e-41ca-bcf7-310e07554447\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"
Jul 07 02:29:17.625132 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.625089    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws56n\" (UniqueName: \"kubernetes.io/projected/ac0f3001-a96e-41ca-bcf7-310e07554447-kube-api-access-ws56n\") pod \"cert-manager-cainjector-559cdd8c87-wb4f9\" (UID: \"ac0f3001-a96e-41ca-bcf7-310e07554447\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"
Jul 07 02:29:17.625344 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.625175    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac0f3001-a96e-41ca-bcf7-310e07554447-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wb4f9\" (UID: \"ac0f3001-a96e-41ca-bcf7-310e07554447\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"
Jul 07 02:29:17.633773 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.633734    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws56n\" (UniqueName: \"kubernetes.io/projected/ac0f3001-a96e-41ca-bcf7-310e07554447-kube-api-access-ws56n\") pod \"cert-manager-cainjector-559cdd8c87-wb4f9\" (UID: \"ac0f3001-a96e-41ca-bcf7-310e07554447\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"
Jul 07 02:29:17.634128 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.634109    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ac0f3001-a96e-41ca-bcf7-310e07554447-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wb4f9\" (UID: \"ac0f3001-a96e-41ca-bcf7-310e07554447\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"
Jul 07 02:29:17.752881 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.752799    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"
Jul 07 02:29:17.896235 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:17.896210    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9"]
Jul 07 02:29:17.898357 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:29:17.898325    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac0f3001_a96e_41ca_bcf7_310e07554447.slice/crio-233eb54e34a1f3edca965a4d8941767b0032fcf993971865d78ca18fba5623b1 WatchSource:0}: Error finding container 233eb54e34a1f3edca965a4d8941767b0032fcf993971865d78ca18fba5623b1: Status 404 returned error can't find the container with id 233eb54e34a1f3edca965a4d8941767b0032fcf993971865d78ca18fba5623b1
Jul 07 02:29:18.333710 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:18.333669    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9" event={"ID":"ac0f3001-a96e-41ca-bcf7-310e07554447","Type":"ContainerStarted","Data":"233eb54e34a1f3edca965a4d8941767b0032fcf993971865d78ca18fba5623b1"}
Jul 07 02:29:21.349098 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:21.349061    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9" event={"ID":"ac0f3001-a96e-41ca-bcf7-310e07554447","Type":"ContainerStarted","Data":"24339e73f3656ab8f9041446bbdb1289ac4cfa9aa582bb55ce49f595b0de783a"}
Jul 07 02:29:21.366772 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:21.366722    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wb4f9" podStartSLOduration=1.9732771150000001 podStartE2EDuration="4.36670211s" totalImagesPullingTime="2.393424995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:21.364544178 +0000 UTC m=+674.007704699" watchObservedRunningTime="2026-07-07 02:29:21.36670211 +0000 UTC m=+674.009862632"
Jul 07 02:29:34.211474 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.211439    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"]
Jul 07 02:29:34.215570 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.215551    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"
Jul 07 02:29:34.218917 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.218896    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 07 02:29:34.218917 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.218904    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 07 02:29:34.219737 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.219723    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5sscc\""
Jul 07 02:29:34.224242 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.224220    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"]
Jul 07 02:29:34.272828 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.272791    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndld\" (UniqueName: \"kubernetes.io/projected/4e33ee66-cd16-488f-a6e1-b332256dbb52-kube-api-access-2ndld\") pod \"openshift-lws-operator-fd8ccff4c-xkhwz\" (UID: \"4e33ee66-cd16-488f-a6e1-b332256dbb52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"
Jul 07 02:29:34.273015 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.272861    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e33ee66-cd16-488f-a6e1-b332256dbb52-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xkhwz\" (UID: \"4e33ee66-cd16-488f-a6e1-b332256dbb52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"
Jul 07 02:29:34.373295 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.373258    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e33ee66-cd16-488f-a6e1-b332256dbb52-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xkhwz\" (UID: \"4e33ee66-cd16-488f-a6e1-b332256dbb52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"
Jul 07 02:29:34.373473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.373335    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndld\" (UniqueName: \"kubernetes.io/projected/4e33ee66-cd16-488f-a6e1-b332256dbb52-kube-api-access-2ndld\") pod \"openshift-lws-operator-fd8ccff4c-xkhwz\" (UID: \"4e33ee66-cd16-488f-a6e1-b332256dbb52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"
Jul 07 02:29:34.373715 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.373692    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4e33ee66-cd16-488f-a6e1-b332256dbb52-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xkhwz\" (UID: \"4e33ee66-cd16-488f-a6e1-b332256dbb52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"
Jul 07 02:29:34.386986 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.386950    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndld\" (UniqueName: \"kubernetes.io/projected/4e33ee66-cd16-488f-a6e1-b332256dbb52-kube-api-access-2ndld\") pod \"openshift-lws-operator-fd8ccff4c-xkhwz\" (UID: \"4e33ee66-cd16-488f-a6e1-b332256dbb52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"
Jul 07 02:29:34.524836 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.524747    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"
Jul 07 02:29:34.664709 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.664674    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz"]
Jul 07 02:29:34.667726 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:29:34.667699    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e33ee66_cd16_488f_a6e1_b332256dbb52.slice/crio-26897394b2b27187d25f1bffae69c9f49a5001704a14ed4475f6becb792160bb WatchSource:0}: Error finding container 26897394b2b27187d25f1bffae69c9f49a5001704a14ed4475f6becb792160bb: Status 404 returned error can't find the container with id 26897394b2b27187d25f1bffae69c9f49a5001704a14ed4475f6becb792160bb
Jul 07 02:29:34.669287 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:34.669272    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:29:35.410102 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:35.410044    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz" event={"ID":"4e33ee66-cd16-488f-a6e1-b332256dbb52","Type":"ContainerStarted","Data":"26897394b2b27187d25f1bffae69c9f49a5001704a14ed4475f6becb792160bb"}
Jul 07 02:29:37.421787 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:37.421752    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz" event={"ID":"4e33ee66-cd16-488f-a6e1-b332256dbb52","Type":"ContainerStarted","Data":"2bd4f45ed9f9a657be7bf40d26e6047b365bf45968221fcddb0f66934a321ee5"}
Jul 07 02:29:37.443455 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:37.443395    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xkhwz" podStartSLOduration=0.76027585 podStartE2EDuration="3.44337312s" totalImagesPullingTime="2.68309727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:37.440835611 +0000 UTC m=+690.083996147" watchObservedRunningTime="2026-07-07 02:29:37.44337312 +0000 UTC m=+690.086533642"
Jul 07 02:29:54.949703 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:54.949667    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"]
Jul 07 02:29:54.953030 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:54.953014    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:54.955635 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:54.955616    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 07 02:29:54.956592 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:54.956574    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 07 02:29:54.956645 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:54.956584    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-lbvft\""
Jul 07 02:29:54.956680 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:54.956584    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 07 02:29:54.965095 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:54.965071    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"]
Jul 07 02:29:55.020940 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.020903    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-metrics-cert\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.021119 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.020959    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsh6j\" (UniqueName: \"kubernetes.io/projected/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-kube-api-access-jsh6j\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.021119 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.020987    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-cert\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.021119 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.021008    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-manager-config\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.121503 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.121469    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsh6j\" (UniqueName: \"kubernetes.io/projected/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-kube-api-access-jsh6j\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.121687 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.121510    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-cert\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.121687 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.121532    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-manager-config\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.121687 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.121568    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-metrics-cert\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.122295 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.122274    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-manager-config\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.124056 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.124030    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-cert\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.124272 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.124251    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-metrics-cert\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.133328 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.133309    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsh6j\" (UniqueName: \"kubernetes.io/projected/f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7-kube-api-access-jsh6j\") pod \"lws-controller-manager-d5cdd5cf7-f6h48\" (UID: \"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7\") " pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.263044 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.262955    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:55.416851 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.416827    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"]
Jul 07 02:29:55.419307 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:29:55.419282    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8ee2e5f_89d3_4a05_bf24_fd3c79fccba7.slice/crio-ba52e7a74ed8a184c8315c1cbb6a37e1378dff992aaf41169cecb528d9f4a057 WatchSource:0}: Error finding container ba52e7a74ed8a184c8315c1cbb6a37e1378dff992aaf41169cecb528d9f4a057: Status 404 returned error can't find the container with id ba52e7a74ed8a184c8315c1cbb6a37e1378dff992aaf41169cecb528d9f4a057
Jul 07 02:29:55.495321 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:55.495276    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48" event={"ID":"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7","Type":"ContainerStarted","Data":"ba52e7a74ed8a184c8315c1cbb6a37e1378dff992aaf41169cecb528d9f4a057"}
Jul 07 02:29:58.510569 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:58.510531    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48" event={"ID":"f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7","Type":"ContainerStarted","Data":"d4cf00ed9e51c6d84d9ab8be6d6c916edf297b76ecd9fa812955342f8fdd14ce"}
Jul 07 02:29:58.511058 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:58.510670    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:29:58.529662 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:29:58.529607    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48" podStartSLOduration=2.076895185 podStartE2EDuration="4.529592216s" totalImagesPullingTime="2.452697031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:29:58.527953452 +0000 UTC m=+711.171113974" watchObservedRunningTime="2026-07-07 02:29:58.529592216 +0000 UTC m=+711.172752737"
Jul 07 02:30:09.517258 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:09.517226    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d5cdd5cf7-f6h48"
Jul 07 02:30:38.103014 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.102979    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"]
Jul 07 02:30:38.105217 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.105176    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.107872 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.107847    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 07 02:30:38.107986 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.107850    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nvhdn\""
Jul 07 02:30:38.108048 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.108019    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 07 02:30:38.108806 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.108784    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 07 02:30:38.108916 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.108870    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 07 02:30:38.119241 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.119214    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"]
Jul 07 02:30:38.289454 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.289423    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1177dde-fbad-4ec5-a787-dcdd8cdbe812-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-g9hbm\" (UID: \"a1177dde-fbad-4ec5-a787-dcdd8cdbe812\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.289454 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.289457    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgq7s\" (UniqueName: \"kubernetes.io/projected/a1177dde-fbad-4ec5-a787-dcdd8cdbe812-kube-api-access-vgq7s\") pod \"kuadrant-console-plugin-85fd4b8b9d-g9hbm\" (UID: \"a1177dde-fbad-4ec5-a787-dcdd8cdbe812\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.289670 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.289492    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1177dde-fbad-4ec5-a787-dcdd8cdbe812-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-g9hbm\" (UID: \"a1177dde-fbad-4ec5-a787-dcdd8cdbe812\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.390456 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.390415    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1177dde-fbad-4ec5-a787-dcdd8cdbe812-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-g9hbm\" (UID: \"a1177dde-fbad-4ec5-a787-dcdd8cdbe812\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.390456 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.390459    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgq7s\" (UniqueName: \"kubernetes.io/projected/a1177dde-fbad-4ec5-a787-dcdd8cdbe812-kube-api-access-vgq7s\") pod \"kuadrant-console-plugin-85fd4b8b9d-g9hbm\" (UID: \"a1177dde-fbad-4ec5-a787-dcdd8cdbe812\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.390693 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.390488    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1177dde-fbad-4ec5-a787-dcdd8cdbe812-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-g9hbm\" (UID: \"a1177dde-fbad-4ec5-a787-dcdd8cdbe812\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.391116 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.391093    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/a1177dde-fbad-4ec5-a787-dcdd8cdbe812-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-g9hbm\" (UID: \"a1177dde-fbad-4ec5-a787-dcdd8cdbe812\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.392955 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.392933    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1177dde-fbad-4ec5-a787-dcdd8cdbe812-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-g9hbm\" (UID: \"a1177dde-fbad-4ec5-a787-dcdd8cdbe812\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.399498 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.399467    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgq7s\" (UniqueName: \"kubernetes.io/projected/a1177dde-fbad-4ec5-a787-dcdd8cdbe812-kube-api-access-vgq7s\") pod \"kuadrant-console-plugin-85fd4b8b9d-g9hbm\" (UID: \"a1177dde-fbad-4ec5-a787-dcdd8cdbe812\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.416216 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.416174    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"
Jul 07 02:30:38.553014 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.552954    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm"]
Jul 07 02:30:38.555570 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:30:38.555543    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1177dde_fbad_4ec5_a787_dcdd8cdbe812.slice/crio-b82160938f161838ab13558ee4019b157dfc30c52550537a2483796f942c4549 WatchSource:0}: Error finding container b82160938f161838ab13558ee4019b157dfc30c52550537a2483796f942c4549: Status 404 returned error can't find the container with id b82160938f161838ab13558ee4019b157dfc30c52550537a2483796f942c4549
Jul 07 02:30:38.682285 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:38.682182    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm" event={"ID":"a1177dde-fbad-4ec5-a787-dcdd8cdbe812","Type":"ContainerStarted","Data":"b82160938f161838ab13558ee4019b157dfc30c52550537a2483796f942c4549"}
Jul 07 02:30:45.716151 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:45.716117    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm" event={"ID":"a1177dde-fbad-4ec5-a787-dcdd8cdbe812","Type":"ContainerStarted","Data":"8bb1f4be6d4d48de6eece943bca0f0032a1706a9d9a04d0696d18956142dbb35"}
Jul 07 02:30:45.733052 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:30:45.732998    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-g9hbm" podStartSLOduration=1.513278487 podStartE2EDuration="7.732982603s" totalImagesPullingTime="6.219704116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:30:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:30:45.731709801 +0000 UTC m=+758.374870323" watchObservedRunningTime="2026-07-07 02:30:45.732982603 +0000 UTC m=+758.376143124"
Jul 07 02:31:21.427740 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.427701    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-47dnz"]
Jul 07 02:31:21.447177 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.447149    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-47dnz"]
Jul 07 02:31:21.447362 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.447260    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-47dnz"
Jul 07 02:31:21.450649 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.450626    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7l9hp\""
Jul 07 02:31:21.554976 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.554939    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8k2\" (UniqueName: \"kubernetes.io/projected/a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f-kube-api-access-4f8k2\") pod \"authorino-b6c89bb85-47dnz\" (UID: \"a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f\") " pod="kuadrant-system/authorino-b6c89bb85-47dnz"
Jul 07 02:31:21.655875 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.655840    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8k2\" (UniqueName: \"kubernetes.io/projected/a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f-kube-api-access-4f8k2\") pod \"authorino-b6c89bb85-47dnz\" (UID: \"a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f\") " pod="kuadrant-system/authorino-b6c89bb85-47dnz"
Jul 07 02:31:21.665869 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.665836    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8k2\" (UniqueName: \"kubernetes.io/projected/a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f-kube-api-access-4f8k2\") pod \"authorino-b6c89bb85-47dnz\" (UID: \"a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f\") " pod="kuadrant-system/authorino-b6c89bb85-47dnz"
Jul 07 02:31:21.757067 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.756973    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-47dnz"
Jul 07 02:31:21.764310 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.764284    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-rs5dk"]
Jul 07 02:31:21.767866 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.767843    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-rs5dk"
Jul 07 02:31:21.779242 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.779216    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-rs5dk"]
Jul 07 02:31:21.857759 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.857726    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k8jf\" (UniqueName: \"kubernetes.io/projected/faf35625-e8fa-46de-b008-1cdc77434096-kube-api-access-6k8jf\") pod \"authorino-8df697c9f-rs5dk\" (UID: \"faf35625-e8fa-46de-b008-1cdc77434096\") " pod="kuadrant-system/authorino-8df697c9f-rs5dk"
Jul 07 02:31:21.896897 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.896865    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-47dnz"]
Jul 07 02:31:21.899992 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:31:21.899962    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5eb72ce_9e64_4f70_af9b_a1b95ff8f33f.slice/crio-be1d3c1ff497da4f964a1a40423b3ae0cece19b4f89557d33708f9298cd18d03 WatchSource:0}: Error finding container be1d3c1ff497da4f964a1a40423b3ae0cece19b4f89557d33708f9298cd18d03: Status 404 returned error can't find the container with id be1d3c1ff497da4f964a1a40423b3ae0cece19b4f89557d33708f9298cd18d03
Jul 07 02:31:21.958522 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.958487    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k8jf\" (UniqueName: \"kubernetes.io/projected/faf35625-e8fa-46de-b008-1cdc77434096-kube-api-access-6k8jf\") pod \"authorino-8df697c9f-rs5dk\" (UID: \"faf35625-e8fa-46de-b008-1cdc77434096\") " pod="kuadrant-system/authorino-8df697c9f-rs5dk"
Jul 07 02:31:21.967767 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:21.967739    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k8jf\" (UniqueName: \"kubernetes.io/projected/faf35625-e8fa-46de-b008-1cdc77434096-kube-api-access-6k8jf\") pod \"authorino-8df697c9f-rs5dk\" (UID: \"faf35625-e8fa-46de-b008-1cdc77434096\") " pod="kuadrant-system/authorino-8df697c9f-rs5dk"
Jul 07 02:31:22.090544 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:22.090441    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-rs5dk"
Jul 07 02:31:22.227220 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:22.224526    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-rs5dk"]
Jul 07 02:31:22.231979 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:31:22.231950    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfaf35625_e8fa_46de_b008_1cdc77434096.slice/crio-93b6ed6aeeb9d84bcd21cd37226236d4699b14917836cd88465d30b8b9eea68f WatchSource:0}: Error finding container 93b6ed6aeeb9d84bcd21cd37226236d4699b14917836cd88465d30b8b9eea68f: Status 404 returned error can't find the container with id 93b6ed6aeeb9d84bcd21cd37226236d4699b14917836cd88465d30b8b9eea68f
Jul 07 02:31:22.881926 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:22.881846    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-rs5dk" event={"ID":"faf35625-e8fa-46de-b008-1cdc77434096","Type":"ContainerStarted","Data":"93b6ed6aeeb9d84bcd21cd37226236d4699b14917836cd88465d30b8b9eea68f"}
Jul 07 02:31:22.884880 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:22.884822    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-47dnz" event={"ID":"a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f","Type":"ContainerStarted","Data":"be1d3c1ff497da4f964a1a40423b3ae0cece19b4f89557d33708f9298cd18d03"}
Jul 07 02:31:24.898107 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:24.898075    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-rs5dk" event={"ID":"faf35625-e8fa-46de-b008-1cdc77434096","Type":"ContainerStarted","Data":"c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71"}
Jul 07 02:31:24.899644 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:24.899621    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-47dnz" event={"ID":"a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f","Type":"ContainerStarted","Data":"ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5"}
Jul 07 02:31:24.913021 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:24.912977    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-rs5dk" podStartSLOduration=1.4123951 podStartE2EDuration="3.912963906s" totalImagesPullingTime="2.500568806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:31:24.912438012 +0000 UTC m=+797.555598534" watchObservedRunningTime="2026-07-07 02:31:24.912963906 +0000 UTC m=+797.556124427"
Jul 07 02:31:24.929097 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:24.929049    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-47dnz" podStartSLOduration=1.107121095 podStartE2EDuration="3.929033875s" totalImagesPullingTime="2.82191278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:31:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:31:24.92648508 +0000 UTC m=+797.569645604" watchObservedRunningTime="2026-07-07 02:31:24.929033875 +0000 UTC m=+797.572194396"
Jul 07 02:31:24.950948 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:24.950912    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-47dnz"]
Jul 07 02:31:26.908538 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:26.908497    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-47dnz" podUID="a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f" containerName="authorino" containerID="cri-o://ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5" gracePeriod=30
Jul 07 02:31:27.158692 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.158635    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-47dnz"
Jul 07 02:31:27.309960 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.309921    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f8k2\" (UniqueName: \"kubernetes.io/projected/a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f-kube-api-access-4f8k2\") pod \"a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f\" (UID: \"a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f\") "
Jul 07 02:31:27.312357 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.312322    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f-kube-api-access-4f8k2" (OuterVolumeSpecName: "kube-api-access-4f8k2") pod "a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f" (UID: "a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f"). InnerVolumeSpecName "kube-api-access-4f8k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:31:27.411430 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.411338    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f8k2\" (UniqueName: \"kubernetes.io/projected/a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f-kube-api-access-4f8k2\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:31:27.913790 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.913755    2543 generic.go:356] "Generic (PLEG): container finished" podID="a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f" containerID="ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5" exitCode=0
Jul 07 02:31:27.914250 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.913803    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-47dnz"
Jul 07 02:31:27.914250 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.913802    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-47dnz" event={"ID":"a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f","Type":"ContainerDied","Data":"ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5"}
Jul 07 02:31:27.914250 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.913917    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-47dnz" event={"ID":"a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f","Type":"ContainerDied","Data":"be1d3c1ff497da4f964a1a40423b3ae0cece19b4f89557d33708f9298cd18d03"}
Jul 07 02:31:27.914250 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.913942    2543 scope.go:117] "RemoveContainer" containerID="ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5"
Jul 07 02:31:27.922760 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.922742    2543 scope.go:117] "RemoveContainer" containerID="ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5"
Jul 07 02:31:27.923008 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:31:27.922988    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5\": container with ID starting with ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5 not found: ID does not exist" containerID="ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5"
Jul 07 02:31:27.923075 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.923020    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5"} err="failed to get container status \"ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5\": rpc error: code = NotFound desc = could not find container \"ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5\": container with ID starting with ffe434351d41d2f71162d5fb6d95310302c02a5b27a9618b664d684185074fc5 not found: ID does not exist"
Jul 07 02:31:27.935980 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.935957    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-47dnz"]
Jul 07 02:31:27.941543 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.941521    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-47dnz"]
Jul 07 02:31:27.979704 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:27.979673    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f" path="/var/lib/kubelet/pods/a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f/volumes"
Jul 07 02:31:47.103231 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.103201    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-kl894"]
Jul 07 02:31:47.103618 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.103529    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f" containerName="authorino"
Jul 07 02:31:47.103618 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.103541    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f" containerName="authorino"
Jul 07 02:31:47.103618 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.103602    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5eb72ce-9e64-4f70-af9b-a1b95ff8f33f" containerName="authorino"
Jul 07 02:31:47.106462 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.106445    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-kl894"
Jul 07 02:31:47.109484 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.109463    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 07 02:31:47.116987 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.116963    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-kl894"]
Jul 07 02:31:47.171498 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.171470    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsk98\" (UniqueName: \"kubernetes.io/projected/99f96f1a-b75a-4956-a8ab-ca32a63f6d26-kube-api-access-fsk98\") pod \"authorino-7c77957575-kl894\" (UID: \"99f96f1a-b75a-4956-a8ab-ca32a63f6d26\") " pod="kuadrant-system/authorino-7c77957575-kl894"
Jul 07 02:31:47.171678 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.171580    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99f96f1a-b75a-4956-a8ab-ca32a63f6d26-tls-cert\") pod \"authorino-7c77957575-kl894\" (UID: \"99f96f1a-b75a-4956-a8ab-ca32a63f6d26\") " pod="kuadrant-system/authorino-7c77957575-kl894"
Jul 07 02:31:47.272111 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.272070    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99f96f1a-b75a-4956-a8ab-ca32a63f6d26-tls-cert\") pod \"authorino-7c77957575-kl894\" (UID: \"99f96f1a-b75a-4956-a8ab-ca32a63f6d26\") " pod="kuadrant-system/authorino-7c77957575-kl894"
Jul 07 02:31:47.272333 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.272126    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsk98\" (UniqueName: \"kubernetes.io/projected/99f96f1a-b75a-4956-a8ab-ca32a63f6d26-kube-api-access-fsk98\") pod \"authorino-7c77957575-kl894\" (UID: \"99f96f1a-b75a-4956-a8ab-ca32a63f6d26\") " pod="kuadrant-system/authorino-7c77957575-kl894"
Jul 07 02:31:47.274709 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.274685    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99f96f1a-b75a-4956-a8ab-ca32a63f6d26-tls-cert\") pod \"authorino-7c77957575-kl894\" (UID: \"99f96f1a-b75a-4956-a8ab-ca32a63f6d26\") " pod="kuadrant-system/authorino-7c77957575-kl894"
Jul 07 02:31:47.281518 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.281489    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsk98\" (UniqueName: \"kubernetes.io/projected/99f96f1a-b75a-4956-a8ab-ca32a63f6d26-kube-api-access-fsk98\") pod \"authorino-7c77957575-kl894\" (UID: \"99f96f1a-b75a-4956-a8ab-ca32a63f6d26\") " pod="kuadrant-system/authorino-7c77957575-kl894"
Jul 07 02:31:47.415784 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.415733    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-kl894"
Jul 07 02:31:47.548238 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:47.548214    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-kl894"]
Jul 07 02:31:47.550250 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:31:47.550220    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99f96f1a_b75a_4956_a8ab_ca32a63f6d26.slice/crio-356b2c2c8cf07a84ffa22ac5e8d494fafe2f0a9f38d071d9e159d77ab7a5c7d5 WatchSource:0}: Error finding container 356b2c2c8cf07a84ffa22ac5e8d494fafe2f0a9f38d071d9e159d77ab7a5c7d5: Status 404 returned error can't find the container with id 356b2c2c8cf07a84ffa22ac5e8d494fafe2f0a9f38d071d9e159d77ab7a5c7d5
Jul 07 02:31:48.004583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:48.004548    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-kl894" event={"ID":"99f96f1a-b75a-4956-a8ab-ca32a63f6d26","Type":"ContainerStarted","Data":"356b2c2c8cf07a84ffa22ac5e8d494fafe2f0a9f38d071d9e159d77ab7a5c7d5"}
Jul 07 02:31:49.010224 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:49.010176    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-kl894" event={"ID":"99f96f1a-b75a-4956-a8ab-ca32a63f6d26","Type":"ContainerStarted","Data":"7956c81b521de09fa4096c31632038ad5eb2eb8485126daa7c230cdb1248021a"}
Jul 07 02:31:49.029389 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:49.029334    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-kl894" podStartSLOduration=1.391171827 podStartE2EDuration="2.029318166s" totalImagesPullingTime="638.146339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:31:49.02727198 +0000 UTC m=+821.670432499" watchObservedRunningTime="2026-07-07 02:31:49.029318166 +0000 UTC m=+821.672478687"
Jul 07 02:31:49.065997 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:49.065955    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-rs5dk"]
Jul 07 02:31:49.066240 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:49.066204    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-rs5dk" podUID="faf35625-e8fa-46de-b008-1cdc77434096" containerName="authorino" containerID="cri-o://c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71" gracePeriod=30
Jul 07 02:31:49.329737 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:49.329708    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-rs5dk"
Jul 07 02:31:49.389295 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:49.389250    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k8jf\" (UniqueName: \"kubernetes.io/projected/faf35625-e8fa-46de-b008-1cdc77434096-kube-api-access-6k8jf\") pod \"faf35625-e8fa-46de-b008-1cdc77434096\" (UID: \"faf35625-e8fa-46de-b008-1cdc77434096\") "
Jul 07 02:31:49.391435 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:49.391406    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf35625-e8fa-46de-b008-1cdc77434096-kube-api-access-6k8jf" (OuterVolumeSpecName: "kube-api-access-6k8jf") pod "faf35625-e8fa-46de-b008-1cdc77434096" (UID: "faf35625-e8fa-46de-b008-1cdc77434096"). InnerVolumeSpecName "kube-api-access-6k8jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:31:49.490671 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:49.490638    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k8jf\" (UniqueName: \"kubernetes.io/projected/faf35625-e8fa-46de-b008-1cdc77434096-kube-api-access-6k8jf\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:31:50.015855 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:50.015823    2543 generic.go:356] "Generic (PLEG): container finished" podID="faf35625-e8fa-46de-b008-1cdc77434096" containerID="c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71" exitCode=0
Jul 07 02:31:50.016298 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:50.015880    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-rs5dk"
Jul 07 02:31:50.016298 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:50.015904    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-rs5dk" event={"ID":"faf35625-e8fa-46de-b008-1cdc77434096","Type":"ContainerDied","Data":"c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71"}
Jul 07 02:31:50.016298 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:50.015951    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-rs5dk" event={"ID":"faf35625-e8fa-46de-b008-1cdc77434096","Type":"ContainerDied","Data":"93b6ed6aeeb9d84bcd21cd37226236d4699b14917836cd88465d30b8b9eea68f"}
Jul 07 02:31:50.016298 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:50.015970    2543 scope.go:117] "RemoveContainer" containerID="c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71"
Jul 07 02:31:50.024913 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:50.024887    2543 scope.go:117] "RemoveContainer" containerID="c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71"
Jul 07 02:31:50.025229 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:31:50.025206    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71\": container with ID starting with c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71 not found: ID does not exist" containerID="c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71"
Jul 07 02:31:50.025314 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:50.025237    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71"} err="failed to get container status \"c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71\": rpc error: code = NotFound desc = could not find container \"c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71\": container with ID starting with c44789ea1c61d0ae7d8d15593a4f826edef494c3e9f6e000197a3409a678ec71 not found: ID does not exist"
Jul 07 02:31:50.048443 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:50.048413    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-rs5dk"]
Jul 07 02:31:50.056854 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:50.056829    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-rs5dk"]
Jul 07 02:31:51.979740 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:51.979708    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="faf35625-e8fa-46de-b008-1cdc77434096" path="/var/lib/kubelet/pods/faf35625-e8fa-46de-b008-1cdc77434096/volumes"
Jul 07 02:31:52.762478 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.762443    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-2xf52"]
Jul 07 02:31:52.762676 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.762632    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52" podUID="102da792-ed38-4de2-ad95-8c2eef04d1b5" containerName="manager" containerID="cri-o://ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363" gracePeriod=10
Jul 07 02:31:52.793294 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.793267    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-2dtqc"]
Jul 07 02:31:52.793577 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.793566    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="faf35625-e8fa-46de-b008-1cdc77434096" containerName="authorino"
Jul 07 02:31:52.793622 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.793580    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="faf35625-e8fa-46de-b008-1cdc77434096" containerName="authorino"
Jul 07 02:31:52.793652 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.793635    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="faf35625-e8fa-46de-b008-1cdc77434096" containerName="authorino"
Jul 07 02:31:52.796416 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.796400    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc"
Jul 07 02:31:52.807473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.807447    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-2dtqc"]
Jul 07 02:31:52.819399 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.819373    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26d8d45b-7585-4e82-ae5a-bc00dac45dba-cert\") pod \"kserve-controller-manager-f8f7445f4-2dtqc\" (UID: \"26d8d45b-7585-4e82-ae5a-bc00dac45dba\") " pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc"
Jul 07 02:31:52.819524 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.819403    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9qn6\" (UniqueName: \"kubernetes.io/projected/26d8d45b-7585-4e82-ae5a-bc00dac45dba-kube-api-access-r9qn6\") pod \"kserve-controller-manager-f8f7445f4-2dtqc\" (UID: \"26d8d45b-7585-4e82-ae5a-bc00dac45dba\") " pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc"
Jul 07 02:31:52.920492 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.920463    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26d8d45b-7585-4e82-ae5a-bc00dac45dba-cert\") pod \"kserve-controller-manager-f8f7445f4-2dtqc\" (UID: \"26d8d45b-7585-4e82-ae5a-bc00dac45dba\") " pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc"
Jul 07 02:31:52.920667 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.920501    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9qn6\" (UniqueName: \"kubernetes.io/projected/26d8d45b-7585-4e82-ae5a-bc00dac45dba-kube-api-access-r9qn6\") pod \"kserve-controller-manager-f8f7445f4-2dtqc\" (UID: \"26d8d45b-7585-4e82-ae5a-bc00dac45dba\") " pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc"
Jul 07 02:31:52.923022 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.923000    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/26d8d45b-7585-4e82-ae5a-bc00dac45dba-cert\") pod \"kserve-controller-manager-f8f7445f4-2dtqc\" (UID: \"26d8d45b-7585-4e82-ae5a-bc00dac45dba\") " pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc"
Jul 07 02:31:52.929576 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:52.929536    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9qn6\" (UniqueName: \"kubernetes.io/projected/26d8d45b-7585-4e82-ae5a-bc00dac45dba-kube-api-access-r9qn6\") pod \"kserve-controller-manager-f8f7445f4-2dtqc\" (UID: \"26d8d45b-7585-4e82-ae5a-bc00dac45dba\") " pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc"
Jul 07 02:31:53.027388 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.027364    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:31:53.030572 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.030543    2543 generic.go:356] "Generic (PLEG): container finished" podID="102da792-ed38-4de2-ad95-8c2eef04d1b5" containerID="ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363" exitCode=0
Jul 07 02:31:53.030694 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.030590    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52"
Jul 07 02:31:53.030694 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.030617    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52" event={"ID":"102da792-ed38-4de2-ad95-8c2eef04d1b5","Type":"ContainerDied","Data":"ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363"}
Jul 07 02:31:53.030694 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.030669    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-2xf52" event={"ID":"102da792-ed38-4de2-ad95-8c2eef04d1b5","Type":"ContainerDied","Data":"97b6915e5ad25200e4662422a51f6715ad8c610c571afd82c8d5262ba5cbc044"}
Jul 07 02:31:53.030694 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.030692    2543 scope.go:117] "RemoveContainer" containerID="ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363"
Jul 07 02:31:53.038674 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.038652    2543 scope.go:117] "RemoveContainer" containerID="ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363"
Jul 07 02:31:53.038982 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:31:53.038961    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363\": container with ID starting with ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363 not found: ID does not exist" containerID="ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363"
Jul 07 02:31:53.039051 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.038993    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363"} err="failed to get container status \"ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363\": rpc error: code = NotFound desc = could not find container \"ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363\": container with ID starting with ebd6fdc2e7af80269447544ef8cc64a363972de96b0cd7e47e101ebb81a40363 not found: ID does not exist"
Jul 07 02:31:53.122348 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.122313    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/102da792-ed38-4de2-ad95-8c2eef04d1b5-cert\") pod \"102da792-ed38-4de2-ad95-8c2eef04d1b5\" (UID: \"102da792-ed38-4de2-ad95-8c2eef04d1b5\") "
Jul 07 02:31:53.122523 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.122372    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jg4tw\" (UniqueName: \"kubernetes.io/projected/102da792-ed38-4de2-ad95-8c2eef04d1b5-kube-api-access-jg4tw\") pod \"102da792-ed38-4de2-ad95-8c2eef04d1b5\" (UID: \"102da792-ed38-4de2-ad95-8c2eef04d1b5\") "
Jul 07 02:31:53.124630 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.124599    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/102da792-ed38-4de2-ad95-8c2eef04d1b5-cert" (OuterVolumeSpecName: "cert") pod "102da792-ed38-4de2-ad95-8c2eef04d1b5" (UID: "102da792-ed38-4de2-ad95-8c2eef04d1b5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:31:53.124739 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.124678    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102da792-ed38-4de2-ad95-8c2eef04d1b5-kube-api-access-jg4tw" (OuterVolumeSpecName: "kube-api-access-jg4tw") pod "102da792-ed38-4de2-ad95-8c2eef04d1b5" (UID: "102da792-ed38-4de2-ad95-8c2eef04d1b5"). InnerVolumeSpecName "kube-api-access-jg4tw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:31:53.149887 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.149855    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc"
Jul 07 02:31:53.223852 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.223769    2543 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/102da792-ed38-4de2-ad95-8c2eef04d1b5-cert\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:31:53.223852 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.223822    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg4tw\" (UniqueName: \"kubernetes.io/projected/102da792-ed38-4de2-ad95-8c2eef04d1b5-kube-api-access-jg4tw\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:31:53.293530 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.293504    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-2dtqc"]
Jul 07 02:31:53.296133 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:31:53.296101    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d8d45b_7585_4e82_ae5a_bc00dac45dba.slice/crio-f89f356aaab32a43fb0a88200cbca3080d7c56b28f9d360e35df4986c96952d4 WatchSource:0}: Error finding container f89f356aaab32a43fb0a88200cbca3080d7c56b28f9d360e35df4986c96952d4: Status 404 returned error can't find the container with id f89f356aaab32a43fb0a88200cbca3080d7c56b28f9d360e35df4986c96952d4
Jul 07 02:31:53.354691 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.352368    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-2xf52"]
Jul 07 02:31:53.359242 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.359215    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-f8f7445f4-2xf52"]
Jul 07 02:31:53.980607 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:53.980525    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102da792-ed38-4de2-ad95-8c2eef04d1b5" path="/var/lib/kubelet/pods/102da792-ed38-4de2-ad95-8c2eef04d1b5/volumes"
Jul 07 02:31:54.037452 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:54.037415    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc" event={"ID":"26d8d45b-7585-4e82-ae5a-bc00dac45dba","Type":"ContainerStarted","Data":"97f8dcdbb9cc583bafef2589492e6d6bb7d6e5641dfd2082f739be080854035a"}
Jul 07 02:31:54.037868 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:54.037472    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc" event={"ID":"26d8d45b-7585-4e82-ae5a-bc00dac45dba","Type":"ContainerStarted","Data":"f89f356aaab32a43fb0a88200cbca3080d7c56b28f9d360e35df4986c96952d4"}
Jul 07 02:31:54.037868 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:54.037549    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc"
Jul 07 02:31:54.055832 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:31:54.055779    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc" podStartSLOduration=1.666647421 podStartE2EDuration="2.055763746s" totalImagesPullingTime="389.116325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:31:54.054132065 +0000 UTC m=+826.697292579" watchObservedRunningTime="2026-07-07 02:31:54.055763746 +0000 UTC m=+826.698924267"
Jul 07 02:32:25.045925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:32:25.045897    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-f8f7445f4-2dtqc"
Jul 07 02:33:07.855388 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:33:07.855296    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:33:07.856367 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:33:07.856347    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:34:28.247743 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.247664    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"]
Jul 07 02:34:28.248256 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.248045    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102da792-ed38-4de2-ad95-8c2eef04d1b5" containerName="manager"
Jul 07 02:34:28.248256 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.248061    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="102da792-ed38-4de2-ad95-8c2eef04d1b5" containerName="manager"
Jul 07 02:34:28.248256 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.248140    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="102da792-ed38-4de2-ad95-8c2eef04d1b5" containerName="manager"
Jul 07 02:34:28.251095 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.251073    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.254563 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.254540    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 02:34:28.254687 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.254565    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 02:34:28.254687 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.254579    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 07 02:34:28.254687 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.254544    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:34:28.262370 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.262350    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"]
Jul 07 02:34:28.324956 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.324915    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.325129 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.324980    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.325129 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.325020    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.325129 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.325044    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlxwh\" (UniqueName: \"kubernetes.io/projected/1455d00e-390d-4adb-a39c-eb0b9648116b-kube-api-access-dlxwh\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.325129 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.325091    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1455d00e-390d-4adb-a39c-eb0b9648116b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.325129 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.325110    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.425884 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.425846    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.426083 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.425893    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.426083 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.425922    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlxwh\" (UniqueName: \"kubernetes.io/projected/1455d00e-390d-4adb-a39c-eb0b9648116b-kube-api-access-dlxwh\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.426083 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.425985    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1455d00e-390d-4adb-a39c-eb0b9648116b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.426083 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.426010    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.426083 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.426035    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.426462 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.426346    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.426462 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.426395    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.426462 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.426446    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.428642 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.428608    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.428745 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.428656    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1455d00e-390d-4adb-a39c-eb0b9648116b-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.438854 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.438827    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlxwh\" (UniqueName: \"kubernetes.io/projected/1455d00e-390d-4adb-a39c-eb0b9648116b-kube-api-access-dlxwh\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.562268 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.562148    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:28.698779 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.698753    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"]
Jul 07 02:34:28.700780 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:34:28.700745    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1455d00e_390d_4adb_a39c_eb0b9648116b.slice/crio-2906856f3ca0fee4f72db18ff4979fc31638be9f63325ede47a3d8ffa213440a WatchSource:0}: Error finding container 2906856f3ca0fee4f72db18ff4979fc31638be9f63325ede47a3d8ffa213440a: Status 404 returned error can't find the container with id 2906856f3ca0fee4f72db18ff4979fc31638be9f63325ede47a3d8ffa213440a
Jul 07 02:34:28.720558 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:28.720529    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql" event={"ID":"1455d00e-390d-4adb-a39c-eb0b9648116b","Type":"ContainerStarted","Data":"2906856f3ca0fee4f72db18ff4979fc31638be9f63325ede47a3d8ffa213440a"}
Jul 07 02:34:30.732293 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:30.732249    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql" event={"ID":"1455d00e-390d-4adb-a39c-eb0b9648116b","Type":"ContainerStarted","Data":"13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2"}
Jul 07 02:34:30.751846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:30.751786    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql" podStartSLOduration=1.766162609 podStartE2EDuration="2.751770528s" totalImagesPullingTime="985.607919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:34:30.750468053 +0000 UTC m=+983.393628580" watchObservedRunningTime="2026-07-07 02:34:30.751770528 +0000 UTC m=+983.394931050"
Jul 07 02:34:31.562641 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:31.562601    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:37.507316 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.507274    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"]
Jul 07 02:34:37.510892 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.510868    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.513378 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.513355    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:34:37.513581 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.513562    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 07 02:34:37.521778 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.521751    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"]
Jul 07 02:34:37.605071 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.605039    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-model-cache\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.605268 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.605085    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkr4m\" (UniqueName: \"kubernetes.io/projected/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kube-api-access-qkr4m\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.605268 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.605177    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-dshm\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.605268 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.605237    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-home\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.605268 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.605257    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tls-certs\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.605427 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.605284    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.605427 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.605349    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706011 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.705974    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706011 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.706013    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706257 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.706056    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-model-cache\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706257 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.706100    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkr4m\" (UniqueName: \"kubernetes.io/projected/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kube-api-access-qkr4m\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706257 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.706162    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-dshm\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706257 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.706219    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-home\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706257 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.706254    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tls-certs\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706475 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.706458    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706524 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.706496    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tmp-dir\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706582 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.706543    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-model-cache\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.706634 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.706619    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-home\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.708740 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.708708    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-dshm\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.708852 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.708818    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tls-certs\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.716041 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.716016    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkr4m\" (UniqueName: \"kubernetes.io/projected/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kube-api-access-qkr4m\") pod \"precise-prefix-cache-test-kserve-5ddf85c88c-s4577\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.822541 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.822457    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:37.980991 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:34:37.980961    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37bb5ce_ba9b_4f61_8bc7_d155cf07ab7e.slice/crio-6729f7d500d81bae009442735c38ffcf059a19470d3e63a99522bf4aee9cfcc5 WatchSource:0}: Error finding container 6729f7d500d81bae009442735c38ffcf059a19470d3e63a99522bf4aee9cfcc5: Status 404 returned error can't find the container with id 6729f7d500d81bae009442735c38ffcf059a19470d3e63a99522bf4aee9cfcc5
Jul 07 02:34:37.981340 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.981321    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"]
Jul 07 02:34:37.982842 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:37.982827    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:34:38.563268 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:38.563227    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:38.579202 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:38.579155    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:38.770815 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:38.770771    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" event={"ID":"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e","Type":"ContainerStarted","Data":"6729f7d500d81bae009442735c38ffcf059a19470d3e63a99522bf4aee9cfcc5"}
Jul 07 02:34:38.785258 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:38.785217    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:41.125383 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:41.125350    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"]
Jul 07 02:34:41.788237 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:41.788125    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" event={"ID":"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e","Type":"ContainerStarted","Data":"c03175d8f3f00a43c123dac054111c7b22bcc7562efd509094bcc4537abb1448"}
Jul 07 02:34:45.808704 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:45.808670    2543 generic.go:356] "Generic (PLEG): container finished" podID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerID="c03175d8f3f00a43c123dac054111c7b22bcc7562efd509094bcc4537abb1448" exitCode=0
Jul 07 02:34:45.808704 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:45.808707    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" event={"ID":"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e","Type":"ContainerDied","Data":"c03175d8f3f00a43c123dac054111c7b22bcc7562efd509094bcc4537abb1448"}
Jul 07 02:34:46.815076 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:46.815036    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" event={"ID":"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e","Type":"ContainerStarted","Data":"60a0c53b686069be8e79ccec3c1342a84b2625207f12c1ade15196d8224b3747"}
Jul 07 02:34:46.822814 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:46.822793    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:46.841560 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:46.841400    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" podStartSLOduration=2.900641899 podStartE2EDuration="9.841372958s" totalImagesPullingTime="3.519579388s" totalInitContainerRuntime="3.421151671s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:34:46.834588945 +0000 UTC m=+999.477749476" watchObservedRunningTime="2026-07-07 02:34:46.841372958 +0000 UTC m=+999.484533482"
Jul 07 02:34:47.822578 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:47.822550    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:47.835145 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:47.835120    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:48.835489 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:48.835458    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:34:56.153406 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.153360    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql" podUID="1455d00e-390d-4adb-a39c-eb0b9648116b" containerName="main" containerID="cri-o://13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2" gracePeriod=45
Jul 07 02:34:56.442329 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.442291    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:56.572909 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.572873    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-model-cache\") pod \"1455d00e-390d-4adb-a39c-eb0b9648116b\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") "
Jul 07 02:34:56.572909 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.572912    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1455d00e-390d-4adb-a39c-eb0b9648116b-tls-certs\") pod \"1455d00e-390d-4adb-a39c-eb0b9648116b\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") "
Jul 07 02:34:56.573135 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.572936    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-tmp-dir\") pod \"1455d00e-390d-4adb-a39c-eb0b9648116b\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") "
Jul 07 02:34:56.573135 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.573028    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlxwh\" (UniqueName: \"kubernetes.io/projected/1455d00e-390d-4adb-a39c-eb0b9648116b-kube-api-access-dlxwh\") pod \"1455d00e-390d-4adb-a39c-eb0b9648116b\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") "
Jul 07 02:34:56.573135 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.573071    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-dshm\") pod \"1455d00e-390d-4adb-a39c-eb0b9648116b\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") "
Jul 07 02:34:56.573135 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.573107    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-home\") pod \"1455d00e-390d-4adb-a39c-eb0b9648116b\" (UID: \"1455d00e-390d-4adb-a39c-eb0b9648116b\") "
Jul 07 02:34:56.573369 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.573214    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1455d00e-390d-4adb-a39c-eb0b9648116b" (UID: "1455d00e-390d-4adb-a39c-eb0b9648116b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:34:56.573369 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.573234    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1455d00e-390d-4adb-a39c-eb0b9648116b" (UID: "1455d00e-390d-4adb-a39c-eb0b9648116b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:34:56.573471 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.573408    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:34:56.573471 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.573433    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:34:56.573471 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.573401    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-home" (OuterVolumeSpecName: "home") pod "1455d00e-390d-4adb-a39c-eb0b9648116b" (UID: "1455d00e-390d-4adb-a39c-eb0b9648116b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:34:56.575446 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.575421    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1455d00e-390d-4adb-a39c-eb0b9648116b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1455d00e-390d-4adb-a39c-eb0b9648116b" (UID: "1455d00e-390d-4adb-a39c-eb0b9648116b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:34:56.575446 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.575422    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-dshm" (OuterVolumeSpecName: "dshm") pod "1455d00e-390d-4adb-a39c-eb0b9648116b" (UID: "1455d00e-390d-4adb-a39c-eb0b9648116b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:34:56.575590 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.575425    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1455d00e-390d-4adb-a39c-eb0b9648116b-kube-api-access-dlxwh" (OuterVolumeSpecName: "kube-api-access-dlxwh") pod "1455d00e-390d-4adb-a39c-eb0b9648116b" (UID: "1455d00e-390d-4adb-a39c-eb0b9648116b"). InnerVolumeSpecName "kube-api-access-dlxwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:34:56.674091 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.674009    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:34:56.674091 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.674040    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1455d00e-390d-4adb-a39c-eb0b9648116b-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:34:56.674091 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.674052    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlxwh\" (UniqueName: \"kubernetes.io/projected/1455d00e-390d-4adb-a39c-eb0b9648116b-kube-api-access-dlxwh\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:34:56.674091 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.674061    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1455d00e-390d-4adb-a39c-eb0b9648116b-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:34:56.861369 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.861337    2543 generic.go:356] "Generic (PLEG): container finished" podID="1455d00e-390d-4adb-a39c-eb0b9648116b" containerID="13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2" exitCode=0
Jul 07 02:34:56.861552 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.861385    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"
Jul 07 02:34:56.861552 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.861414    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql" event={"ID":"1455d00e-390d-4adb-a39c-eb0b9648116b","Type":"ContainerDied","Data":"13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2"}
Jul 07 02:34:56.861552 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.861457    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql" event={"ID":"1455d00e-390d-4adb-a39c-eb0b9648116b","Type":"ContainerDied","Data":"2906856f3ca0fee4f72db18ff4979fc31638be9f63325ede47a3d8ffa213440a"}
Jul 07 02:34:56.861552 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.861478    2543 scope.go:117] "RemoveContainer" containerID="13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2"
Jul 07 02:34:56.870740 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.870721    2543 scope.go:117] "RemoveContainer" containerID="13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2"
Jul 07 02:34:56.871001 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:34:56.870984    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2\": container with ID starting with 13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2 not found: ID does not exist" containerID="13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2"
Jul 07 02:34:56.871053 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.871009    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2"} err="failed to get container status \"13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2\": rpc error: code = NotFound desc = could not find container \"13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2\": container with ID starting with 13735797641c6c336bea513cfec31ba17c87642699b2edb6938e4cb633e1b3f2 not found: ID does not exist"
Jul 07 02:34:56.882790 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.882760    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"]
Jul 07 02:34:56.886847 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:56.886825    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc5x74ql"]
Jul 07 02:34:57.980981 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:34:57.980934    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1455d00e-390d-4adb-a39c-eb0b9648116b" path="/var/lib/kubelet/pods/1455d00e-390d-4adb-a39c-eb0b9648116b/volumes"
Jul 07 02:35:04.379522 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.379485    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"]
Jul 07 02:35:04.380007 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.379804    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1455d00e-390d-4adb-a39c-eb0b9648116b" containerName="main"
Jul 07 02:35:04.380007 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.379814    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1455d00e-390d-4adb-a39c-eb0b9648116b" containerName="main"
Jul 07 02:35:04.380007 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.379868    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="1455d00e-390d-4adb-a39c-eb0b9648116b" containerName="main"
Jul 07 02:35:04.381766 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.381749    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.384716 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.384698    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 07 02:35:04.384820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.384699    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-zq6fc\""
Jul 07 02:35:04.396650 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.396627    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"]
Jul 07 02:35:04.543718 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.543683    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.543894 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.543733    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.543894 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.543768    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.543894 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.543816    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.543894 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.543857    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.543894 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.543882    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7x5\" (UniqueName: \"kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-kube-api-access-nm7x5\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.544055 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.543902    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.544055 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.543922    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.544055 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.543997    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.644628 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.644535    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.644628 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.644592    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.644628 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.644619    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.644903 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.644654    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.644903 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.644676    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.644903 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.644766    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7x5\" (UniqueName: \"kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-kube-api-access-nm7x5\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.644903 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.644825    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.644903 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.644857    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.645143 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.644922    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.645231 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.645137    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.645294 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.645246    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.645294 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.645280    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.645413 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.645387    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.645547 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.645507    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.647337 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.647310    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.647516 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.647500    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.655254 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.655157    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.655488 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.655471    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7x5\" (UniqueName: \"kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-kube-api-access-nm7x5\") pod \"router-gateway-1-openshift-default-54c47cdfc7-6cn2l\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.692111 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.692071    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:04.843984 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.843960    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"]
Jul 07 02:35:04.898797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:04.898700    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" event={"ID":"61609f51-bd34-49c2-89b2-b7ee16ba35c4","Type":"ContainerStarted","Data":"38d5f78842096861c990be66a6f41c4b570c048d5eea2851a1048d151ffa680a"}
Jul 07 02:35:10.213675 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:10.213629    2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 07 02:35:10.213937 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:10.213741    2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 07 02:35:10.213937 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:10.213774    2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 07 02:35:10.926937 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:10.926903    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" event={"ID":"61609f51-bd34-49c2-89b2-b7ee16ba35c4","Type":"ContainerStarted","Data":"465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b"}
Jul 07 02:35:10.949132 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:10.949080    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" podStartSLOduration=1.5890886530000001 podStartE2EDuration="6.949064429s" totalImagesPullingTime="5.359975776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:35:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:35:10.947139392 +0000 UTC m=+1023.590299916" watchObservedRunningTime="2026-07-07 02:35:10.949064429 +0000 UTC m=+1023.592224951"
Jul 07 02:35:11.692454 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:11.692415    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:11.697311 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:11.697286    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:11.931545 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:11.931510    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:11.932562 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:11.932542    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:35:24.948534 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:24.948495    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"]
Jul 07 02:35:24.951115 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:24.951098    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:24.953772 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:24.953751    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 07 02:35:24.963058 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:24.963037    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"]
Jul 07 02:35:25.020799 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.020767    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tmp-dir\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.020799 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.020805    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d6jn\" (UniqueName: \"kubernetes.io/projected/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kube-api-access-9d6jn\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.021051 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.020825    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kserve-provision-location\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.021051 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.020878    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-dshm\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.021051 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.020923    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tls-certs\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.021051 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.020975    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-home\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.021051 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.021026    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-model-cache\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.121491 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.121455    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tmp-dir\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.121491 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.121494    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d6jn\" (UniqueName: \"kubernetes.io/projected/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kube-api-access-9d6jn\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.121733 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.121521    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kserve-provision-location\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.121733 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.121547    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-dshm\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.121733 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.121572    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tls-certs\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.121733 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.121624    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-home\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.121733 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.121660    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-model-cache\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.121966 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.121919    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tmp-dir\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.122019 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.121981    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kserve-provision-location\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.122019 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.121995    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-model-cache\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.122126 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.122101    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-home\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.124115 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.124091    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-dshm\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.124261 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.124240    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tls-certs\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.129772 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.129747    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d6jn\" (UniqueName: \"kubernetes.io/projected/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kube-api-access-9d6jn\") pod \"auth-enabled-test-kserve-6cb4578865-5g826\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.253615 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.253534    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"]
Jul 07 02:35:25.256426 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.256409    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.259303 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.259280    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-ntnqv\""
Jul 07 02:35:25.260828 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.260803    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:35:25.267961 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.267932    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"]
Jul 07 02:35:25.323632 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.323541    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.323632 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.323581    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.323632 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.323608    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8414e855-53b4-4e96-bfa5-27e87bc7621a-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.323835 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.323708    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f2rj\" (UniqueName: \"kubernetes.io/projected/8414e855-53b4-4e96-bfa5-27e87bc7621a-kube-api-access-7f2rj\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.407146 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.407118    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"]
Jul 07 02:35:25.408855 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:35:25.408829    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e893e6c_5d2d_4b8e_9031_ff890cc1b0f9.slice/crio-4e1ab9f603c8368aec3e50609c86845d56cb7913186f6a39563c6f2ea2a0943e WatchSource:0}: Error finding container 4e1ab9f603c8368aec3e50609c86845d56cb7913186f6a39563c6f2ea2a0943e: Status 404 returned error can't find the container with id 4e1ab9f603c8368aec3e50609c86845d56cb7913186f6a39563c6f2ea2a0943e
Jul 07 02:35:25.424769 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.424750    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f2rj\" (UniqueName: \"kubernetes.io/projected/8414e855-53b4-4e96-bfa5-27e87bc7621a-kube-api-access-7f2rj\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.424830 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.424813    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.424873 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.424850    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.424908 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.424875    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8414e855-53b4-4e96-bfa5-27e87bc7621a-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.425320 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.425298    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.425320 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.425313    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.427944 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.427922    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8414e855-53b4-4e96-bfa5-27e87bc7621a-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.437772 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.437750    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f2rj\" (UniqueName: \"kubernetes.io/projected/8414e855-53b4-4e96-bfa5-27e87bc7621a-kube-api-access-7f2rj\") pod \"auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.566647 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.566560    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:25.711248 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.711215    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"]
Jul 07 02:35:25.713262 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:35:25.713231    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8414e855_53b4_4e96_bfa5_27e87bc7621a.slice/crio-88098412ec6ca4a90c9271b9a544420881c71a6f1eaaef1446ab5edba2505966 WatchSource:0}: Error finding container 88098412ec6ca4a90c9271b9a544420881c71a6f1eaaef1446ab5edba2505966: Status 404 returned error can't find the container with id 88098412ec6ca4a90c9271b9a544420881c71a6f1eaaef1446ab5edba2505966
Jul 07 02:35:25.995553 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.995514    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk" event={"ID":"8414e855-53b4-4e96-bfa5-27e87bc7621a","Type":"ContainerStarted","Data":"669faf0b3ecac4f5fd5195974acb515d820b34f2a89916628d7b66be29caffc0"}
Jul 07 02:35:25.996039 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.995580    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk" event={"ID":"8414e855-53b4-4e96-bfa5-27e87bc7621a","Type":"ContainerStarted","Data":"88098412ec6ca4a90c9271b9a544420881c71a6f1eaaef1446ab5edba2505966"}
Jul 07 02:35:25.997454 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.997424    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" event={"ID":"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9","Type":"ContainerStarted","Data":"df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c"}
Jul 07 02:35:25.997581 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:25.997478    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" event={"ID":"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9","Type":"ContainerStarted","Data":"4e1ab9f603c8368aec3e50609c86845d56cb7913186f6a39563c6f2ea2a0943e"}
Jul 07 02:35:27.003796 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:27.003760    2543 generic.go:356] "Generic (PLEG): container finished" podID="8414e855-53b4-4e96-bfa5-27e87bc7621a" containerID="669faf0b3ecac4f5fd5195974acb515d820b34f2a89916628d7b66be29caffc0" exitCode=0
Jul 07 02:35:27.004222 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:27.003846    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk" event={"ID":"8414e855-53b4-4e96-bfa5-27e87bc7621a","Type":"ContainerDied","Data":"669faf0b3ecac4f5fd5195974acb515d820b34f2a89916628d7b66be29caffc0"}
Jul 07 02:35:29.016936 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:29.016904    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk" event={"ID":"8414e855-53b4-4e96-bfa5-27e87bc7621a","Type":"ContainerStarted","Data":"bb414af5c553c33eb6a55b32fef9a42c7161c52999147d924991225118d18762"}
Jul 07 02:35:29.017363 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:29.017041    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:35:29.042039 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:29.041984    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk" podStartSLOduration=1.921163746 podStartE2EDuration="4.041966449s" totalImagesPullingTime="1.376287772s" totalInitContainerRuntime="744.514931ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:35:29.040208351 +0000 UTC m=+1041.683368874" watchObservedRunningTime="2026-07-07 02:35:29.041966449 +0000 UTC m=+1041.685126971"
Jul 07 02:35:30.023551 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:30.023521    2543 generic.go:356] "Generic (PLEG): container finished" podID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerID="df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c" exitCode=0
Jul 07 02:35:30.023932 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:30.023597    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" event={"ID":"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9","Type":"ContainerDied","Data":"df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c"}
Jul 07 02:35:50.214847 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:35:50.214813    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"]
Jul 07 02:36:00.025669 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:00.025585    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:36:05.238558 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:05.238485    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="main" containerID="cri-o://60a0c53b686069be8e79ccec3c1342a84b2625207f12c1ade15196d8224b3747" gracePeriod=45
Jul 07 02:36:05.836610 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:05.836570    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 07 02:36:06.836582 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:06.836532    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 07 02:36:06.837056 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:06.836692    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:36:07.836639 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:07.836591    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 07 02:36:08.836282 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:08.836224    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 07 02:36:09.836207 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:09.836158    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": dial tcp 10.134.0.32:8000: connect: connection refused"
Jul 07 02:36:10.238066 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:10.238032    2543 generic.go:356] "Generic (PLEG): container finished" podID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerID="60a0c53b686069be8e79ccec3c1342a84b2625207f12c1ade15196d8224b3747" exitCode=0
Jul 07 02:36:10.238253 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:10.238120    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" event={"ID":"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e","Type":"ContainerDied","Data":"60a0c53b686069be8e79ccec3c1342a84b2625207f12c1ade15196d8224b3747"}
Jul 07 02:36:11.290102 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.290078    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:36:11.374172 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374136    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tls-certs\") pod \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") "
Jul 07 02:36:11.374367 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374180    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tmp-dir\") pod \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") "
Jul 07 02:36:11.374367 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374243    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qkr4m\" (UniqueName: \"kubernetes.io/projected/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kube-api-access-qkr4m\") pod \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") "
Jul 07 02:36:11.374367 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374275    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-home\") pod \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") "
Jul 07 02:36:11.374367 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374313    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-dshm\") pod \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") "
Jul 07 02:36:11.374367 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374351    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kserve-provision-location\") pod \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") "
Jul 07 02:36:11.374614 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374419    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-model-cache\") pod \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\" (UID: \"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e\") "
Jul 07 02:36:11.374614 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374487    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" (UID: "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:11.374614 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374574    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-home" (OuterVolumeSpecName: "home") pod "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" (UID: "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:11.374764 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374696    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-model-cache" (OuterVolumeSpecName: "model-cache") pod "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" (UID: "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:11.374818 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374772    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:36:11.374818 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.374793    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:36:11.376603 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.376568    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-dshm" (OuterVolumeSpecName: "dshm") pod "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" (UID: "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:11.376893 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.376869    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" (UID: "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:36:11.377206 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.377169    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kube-api-access-qkr4m" (OuterVolumeSpecName: "kube-api-access-qkr4m") pod "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" (UID: "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e"). InnerVolumeSpecName "kube-api-access-qkr4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:36:11.427339 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.427284    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" (UID: "d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:36:11.476036 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.476005    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:36:11.476036 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.476030    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:36:11.476036 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.476040    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:36:11.476276 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.476050    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:36:11.476276 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.476058    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkr4m\" (UniqueName: \"kubernetes.io/projected/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e-kube-api-access-qkr4m\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:36:11.836624 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:11.836574    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded"
Jul 07 02:36:12.248846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:12.248764    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"
Jul 07 02:36:12.248846 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:12.248796    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577" event={"ID":"d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e","Type":"ContainerDied","Data":"6729f7d500d81bae009442735c38ffcf059a19470d3e63a99522bf4aee9cfcc5"}
Jul 07 02:36:12.249076 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:12.248872    2543 scope.go:117] "RemoveContainer" containerID="60a0c53b686069be8e79ccec3c1342a84b2625207f12c1ade15196d8224b3747"
Jul 07 02:36:12.250934 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:12.250910    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" event={"ID":"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9","Type":"ContainerStarted","Data":"7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738"}
Jul 07 02:36:12.257700 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:12.257680    2543 scope.go:117] "RemoveContainer" containerID="c03175d8f3f00a43c123dac054111c7b22bcc7562efd509094bcc4537abb1448"
Jul 07 02:36:12.261178 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:12.261156    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:36:12.272317 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:12.272274    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podStartSLOduration=2.985448693 podStartE2EDuration="48.272231851s" totalImagesPullingTime="41.592319739s" totalInitContainerRuntime="3.694463419s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:36:12.271755946 +0000 UTC m=+1084.914916463" watchObservedRunningTime="2026-07-07 02:36:12.272231851 +0000 UTC m=+1084.915392373"
Jul 07 02:36:12.286287 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:12.286253    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"]
Jul 07 02:36:12.292519 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:12.292489    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-5ddf85c88c-s4577"]
Jul 07 02:36:13.980970 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:13.980940    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" path="/var/lib/kubelet/pods/d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e/volumes"
Jul 07 02:36:15.261657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:15.261632    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:36:15.263343 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:15.263312    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:36:25.261365 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:25.261318    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:36:35.261827 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:35.261783    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:36:45.262236 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:45.262168    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:36:55.262145 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:36:55.262086    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:37:05.262406 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:37:05.262352    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:37:15.261841 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:37:15.261790    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:37:25.262004 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:37:25.261960    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:37:35.261952 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:37:35.261846    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:37:45.262289 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:37:45.262245    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:37:55.261809 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:37:55.261763    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:38:05.271733 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:05.271702    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:38:05.279754 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:05.279724    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:38:07.884860 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:07.884829    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:38:07.885857 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:07.885833    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:38:09.699506 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:09.699472    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"]
Jul 07 02:38:09.702880 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:09.702837    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"]
Jul 07 02:38:09.712428 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:38:09.712388    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:38:09.712428 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:38:09.712428 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 02:38:09.712428 ip-10-0-132-101 kubenswrapper[2543]:  > containerID="bb414af5c553c33eb6a55b32fef9a42c7161c52999147d924991225118d18762" cmd=["/bin/sleep","15"]
Jul 07 02:38:09.712679 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:09.712457    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk" message="" err=<
Jul 07 02:38:09.712679 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:38:09.712679 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 02:38:09.712679 ip-10-0-132-101 kubenswrapper[2543]:  >
Jul 07 02:38:09.712679 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:38:09.712473    2543 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:38:09.712679 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:38:09.712679 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 02:38:09.712679 ip-10-0-132-101 kubenswrapper[2543]:  > pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk" podUID="8414e855-53b4-4e96-bfa5-27e87bc7621a" containerName="main" containerID="cri-o://bb414af5c553c33eb6a55b32fef9a42c7161c52999147d924991225118d18762"
Jul 07 02:38:09.712679 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:09.712529    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk" podUID="8414e855-53b4-4e96-bfa5-27e87bc7621a" containerName="main" containerID="cri-o://bb414af5c553c33eb6a55b32fef9a42c7161c52999147d924991225118d18762" gracePeriod=60
Jul 07 02:38:09.833204 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:09.833164    2543 generic.go:356] "Generic (PLEG): container finished" podID="8414e855-53b4-4e96-bfa5-27e87bc7621a" containerID="bb414af5c553c33eb6a55b32fef9a42c7161c52999147d924991225118d18762" exitCode=0
Jul 07 02:38:09.833315 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:09.833224    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk" event={"ID":"8414e855-53b4-4e96-bfa5-27e87bc7621a","Type":"ContainerDied","Data":"bb414af5c553c33eb6a55b32fef9a42c7161c52999147d924991225118d18762"}
Jul 07 02:38:09.956398 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:09.956340    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:38:10.047411 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.047386    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-tokenizer-uds\") pod \"8414e855-53b4-4e96-bfa5-27e87bc7621a\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") "
Jul 07 02:38:10.047607 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.047436    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f2rj\" (UniqueName: \"kubernetes.io/projected/8414e855-53b4-4e96-bfa5-27e87bc7621a-kube-api-access-7f2rj\") pod \"8414e855-53b4-4e96-bfa5-27e87bc7621a\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") "
Jul 07 02:38:10.047607 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.047466    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-kserve-provision-location\") pod \"8414e855-53b4-4e96-bfa5-27e87bc7621a\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") "
Jul 07 02:38:10.047607 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.047487    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8414e855-53b4-4e96-bfa5-27e87bc7621a-tls-certs\") pod \"8414e855-53b4-4e96-bfa5-27e87bc7621a\" (UID: \"8414e855-53b4-4e96-bfa5-27e87bc7621a\") "
Jul 07 02:38:10.047788 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.047654    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8414e855-53b4-4e96-bfa5-27e87bc7621a" (UID: "8414e855-53b4-4e96-bfa5-27e87bc7621a"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:38:10.048253 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.048171    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8414e855-53b4-4e96-bfa5-27e87bc7621a" (UID: "8414e855-53b4-4e96-bfa5-27e87bc7621a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:38:10.049823 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.049803    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8414e855-53b4-4e96-bfa5-27e87bc7621a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8414e855-53b4-4e96-bfa5-27e87bc7621a" (UID: "8414e855-53b4-4e96-bfa5-27e87bc7621a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:38:10.049911 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.049841    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8414e855-53b4-4e96-bfa5-27e87bc7621a-kube-api-access-7f2rj" (OuterVolumeSpecName: "kube-api-access-7f2rj") pod "8414e855-53b4-4e96-bfa5-27e87bc7621a" (UID: "8414e855-53b4-4e96-bfa5-27e87bc7621a"). InnerVolumeSpecName "kube-api-access-7f2rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:38:10.148593 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.148541    2543 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-tokenizer-uds\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:10.148593 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.148585    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f2rj\" (UniqueName: \"kubernetes.io/projected/8414e855-53b4-4e96-bfa5-27e87bc7621a-kube-api-access-7f2rj\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:10.148593 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.148596    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8414e855-53b4-4e96-bfa5-27e87bc7621a-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:10.148593 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.148606    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8414e855-53b4-4e96-bfa5-27e87bc7621a-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:10.839274 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.839245    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"
Jul 07 02:38:10.839782 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.839295    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk" event={"ID":"8414e855-53b4-4e96-bfa5-27e87bc7621a","Type":"ContainerDied","Data":"88098412ec6ca4a90c9271b9a544420881c71a6f1eaaef1446ab5edba2505966"}
Jul 07 02:38:10.839782 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.839354    2543 scope.go:117] "RemoveContainer" containerID="bb414af5c553c33eb6a55b32fef9a42c7161c52999147d924991225118d18762"
Jul 07 02:38:10.848876 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.848853    2543 scope.go:117] "RemoveContainer" containerID="669faf0b3ecac4f5fd5195974acb515d820b34f2a89916628d7b66be29caffc0"
Jul 07 02:38:10.861932 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.861897    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"]
Jul 07 02:38:10.865631 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:10.865609    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-579c77b84f-6p4wk"]
Jul 07 02:38:11.980096 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:11.980062    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8414e855-53b4-4e96-bfa5-27e87bc7621a" path="/var/lib/kubelet/pods/8414e855-53b4-4e96-bfa5-27e87bc7621a/volumes"
Jul 07 02:38:24.729456 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:24.729412    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" containerID="cri-o://7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738" gracePeriod=45
Jul 07 02:38:25.272253 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.272216    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": dial tcp 10.134.0.34:8000: connect: connection refused"
Jul 07 02:38:25.484963 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.484938    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:38:25.583761 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.583677    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-model-cache\") pod \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") "
Jul 07 02:38:25.583761 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.583719    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-home\") pod \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") "
Jul 07 02:38:25.583761 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.583758    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d6jn\" (UniqueName: \"kubernetes.io/projected/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kube-api-access-9d6jn\") pod \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") "
Jul 07 02:38:25.584058 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.583775    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tls-certs\") pod \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") "
Jul 07 02:38:25.584058 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.583821    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-dshm\") pod \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") "
Jul 07 02:38:25.584058 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.583849    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kserve-provision-location\") pod \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") "
Jul 07 02:38:25.584058 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.583919    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tmp-dir\") pod \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\" (UID: \"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9\") "
Jul 07 02:38:25.584058 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.583998    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" (UID: "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:38:25.584361 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.584277    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:25.584569 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.584544    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-home" (OuterVolumeSpecName: "home") pod "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" (UID: "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:38:25.586284 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.586253    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" (UID: "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:38:25.586417 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.586393    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-dshm" (OuterVolumeSpecName: "dshm") pod "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" (UID: "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:38:25.586481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.586455    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kube-api-access-9d6jn" (OuterVolumeSpecName: "kube-api-access-9d6jn") pod "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" (UID: "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9"). InnerVolumeSpecName "kube-api-access-9d6jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:38:25.603559 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.603508    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" (UID: "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:38:25.646651 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.646615    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" (UID: "0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:38:25.684872 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.684840    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d6jn\" (UniqueName: \"kubernetes.io/projected/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kube-api-access-9d6jn\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:25.684872 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.684867    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:25.684872 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.684877    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:25.685080 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.684885    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:25.685080 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.684895    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:25.685080 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.684904    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:38:25.909323 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.909286    2543 generic.go:356] "Generic (PLEG): container finished" podID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerID="7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738" exitCode=0
Jul 07 02:38:25.909742 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.909346    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"
Jul 07 02:38:25.909742 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.909367    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" event={"ID":"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9","Type":"ContainerDied","Data":"7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738"}
Jul 07 02:38:25.909742 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.909431    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826" event={"ID":"0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9","Type":"ContainerDied","Data":"4e1ab9f603c8368aec3e50609c86845d56cb7913186f6a39563c6f2ea2a0943e"}
Jul 07 02:38:25.909742 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.909462    2543 scope.go:117] "RemoveContainer" containerID="7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738"
Jul 07 02:38:25.918656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.918640    2543 scope.go:117] "RemoveContainer" containerID="df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c"
Jul 07 02:38:25.926619 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.926604    2543 scope.go:117] "RemoveContainer" containerID="7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738"
Jul 07 02:38:25.926884 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:38:25.926868    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738\": container with ID starting with 7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738 not found: ID does not exist" containerID="7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738"
Jul 07 02:38:25.926924 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.926896    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738"} err="failed to get container status \"7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738\": rpc error: code = NotFound desc = could not find container \"7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738\": container with ID starting with 7e5b65f9368aa684b1f5bd99f6ee09c5f0afee2b6124b34ec3f65e1f5fdc0738 not found: ID does not exist"
Jul 07 02:38:25.926924 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.926914    2543 scope.go:117] "RemoveContainer" containerID="df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c"
Jul 07 02:38:25.927145 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:38:25.927125    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c\": container with ID starting with df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c not found: ID does not exist" containerID="df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c"
Jul 07 02:38:25.927179 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.927153    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c"} err="failed to get container status \"df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c\": rpc error: code = NotFound desc = could not find container \"df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c\": container with ID starting with df8014420fadb8c02a3d134817a87e3364c1ec66c14b090d600057ce486e9f6c not found: ID does not exist"
Jul 07 02:38:25.935473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.935451    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"]
Jul 07 02:38:25.939178 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.939154    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-6cb4578865-5g826"]
Jul 07 02:38:25.981673 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:25.981640    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" path="/var/lib/kubelet/pods/0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9/volumes"
Jul 07 02:38:33.335626 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335591    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"]
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335908    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="storage-initializer"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335918    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="storage-initializer"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335927    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335934    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335950    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="storage-initializer"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335955    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="storage-initializer"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335961    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8414e855-53b4-4e96-bfa5-27e87bc7621a" containerName="storage-initializer"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335966    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8414e855-53b4-4e96-bfa5-27e87bc7621a" containerName="storage-initializer"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335978    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="main"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335983    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="main"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335987    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8414e855-53b4-4e96-bfa5-27e87bc7621a" containerName="main"
Jul 07 02:38:33.336020 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.335993    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8414e855-53b4-4e96-bfa5-27e87bc7621a" containerName="main"
Jul 07 02:38:33.336439 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.336040    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8414e855-53b4-4e96-bfa5-27e87bc7621a" containerName="main"
Jul 07 02:38:33.336439 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.336048    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e893e6c-5d2d-4b8e-9031-ff890cc1b0f9" containerName="main"
Jul 07 02:38:33.336439 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.336053    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d37bb5ce-ba9b-4f61-8bc7-d155cf07ab7e" containerName="main"
Jul 07 02:38:33.340694 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.340675    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.346576 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.346553    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 07 02:38:33.346674 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.346594    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:38:33.356223 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.356163    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"]
Jul 07 02:38:33.453277 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.453236    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xqfd\" (UniqueName: \"kubernetes.io/projected/3a29a134-cf45-4008-a645-f15a6a0a0f22-kube-api-access-4xqfd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.453469 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.453295    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.453469 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.453364    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.453469 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.453390    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.453469 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.453417    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.453469 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.453462    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a29a134-cf45-4008-a645-f15a6a0a0f22-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.554514 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.554472    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a29a134-cf45-4008-a645-f15a6a0a0f22-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.554514 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.554518    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xqfd\" (UniqueName: \"kubernetes.io/projected/3a29a134-cf45-4008-a645-f15a6a0a0f22-kube-api-access-4xqfd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.554752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.554570    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.554752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.554627    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.554752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.554652    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.554752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.554681    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.555174 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.555143    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.555320 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.555230    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.555320 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.555296    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.557442 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.557417    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.557759 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.557738    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a29a134-cf45-4008-a645-f15a6a0a0f22-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.568911 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.568877    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xqfd\" (UniqueName: \"kubernetes.io/projected/3a29a134-cf45-4008-a645-f15a6a0a0f22-kube-api-access-4xqfd\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") " pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.651713 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.651667    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:33.796599 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.796574    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"]
Jul 07 02:38:33.798578 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:38:33.798556    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a29a134_cf45_4008_a645_f15a6a0a0f22.slice/crio-dd621f9667d560440790d06285b8c1717fcb8921330b95f52dd2f44a60578c69 WatchSource:0}: Error finding container dd621f9667d560440790d06285b8c1717fcb8921330b95f52dd2f44a60578c69: Status 404 returned error can't find the container with id dd621f9667d560440790d06285b8c1717fcb8921330b95f52dd2f44a60578c69
Jul 07 02:38:33.946696 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.946608    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d" event={"ID":"3a29a134-cf45-4008-a645-f15a6a0a0f22","Type":"ContainerStarted","Data":"f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf"}
Jul 07 02:38:33.946696 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.946677    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d" event={"ID":"3a29a134-cf45-4008-a645-f15a6a0a0f22","Type":"ContainerStarted","Data":"dd621f9667d560440790d06285b8c1717fcb8921330b95f52dd2f44a60578c69"}
Jul 07 02:38:33.966493 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:33.966440    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d" podStartSLOduration=0.966420641 podStartE2EDuration="966.420641ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:38:33.965571701 +0000 UTC m=+1226.608732227" watchObservedRunningTime="2026-07-07 02:38:33.966420641 +0000 UTC m=+1226.609581162"
Jul 07 02:38:34.652657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:34.652610    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:43.652275 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:43.652228    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:43.664933 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:43.664908    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:44.002550 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:44.002454    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:38:51.732473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.732438    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"]
Jul 07 02:38:51.735958 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.735940    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.738533 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.738508    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:38:51.738658 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.738615    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 07 02:38:51.755991 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.754601    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"]
Jul 07 02:38:51.800473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.800434    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.800473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.800479    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-dshm\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.800737 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.800497    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrls\" (UniqueName: \"kubernetes.io/projected/ec05c28b-6542-423b-be5a-13387203eff7-kube-api-access-qrrls\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.800737 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.800619    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-tmp-dir\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.800737 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.800680    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-home\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.800737 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.800705    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-model-cache\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.800737 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.800729    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec05c28b-6542-423b-be5a-13387203eff7-tls-certs\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902108 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902066    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-home\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902108 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902108    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-model-cache\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902389 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902126    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec05c28b-6542-423b-be5a-13387203eff7-tls-certs\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902389 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902161    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902389 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902225    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-dshm\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902389 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902242    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrls\" (UniqueName: \"kubernetes.io/projected/ec05c28b-6542-423b-be5a-13387203eff7-kube-api-access-qrrls\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902389 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902303    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-tmp-dir\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902664 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902541    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-home\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902664 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902602    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-model-cache\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902740 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902663    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.902740 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.902687    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-tmp-dir\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.904604 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.904579    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-dshm\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.904897 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.904880    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec05c28b-6542-423b-be5a-13387203eff7-tls-certs\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.912021 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.911995    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrls\" (UniqueName: \"kubernetes.io/projected/ec05c28b-6542-423b-be5a-13387203eff7-kube-api-access-qrrls\") pod \"auth-invalid-token-test-kserve-666bbbb8df-6gg4b\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:51.969367 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.969335    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"]
Jul 07 02:38:51.973128 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.973102    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:51.975712 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.975685    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-txrsb\""
Jul 07 02:38:51.988576 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:51.988510    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"]
Jul 07 02:38:52.003214 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.003149    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k46x\" (UniqueName: \"kubernetes.io/projected/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kube-api-access-6k46x\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.003618 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.003247    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.003618 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.003291    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.003618 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.003394    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.046498 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.046460    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:38:52.104659 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.104629    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.104817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.104691    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k46x\" (UniqueName: \"kubernetes.io/projected/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kube-api-access-6k46x\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.104817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.104722    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.104817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.104750    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.105127 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.105086    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.105233 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.105094    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.107815 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.107784    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.114690 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.114609    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k46x\" (UniqueName: \"kubernetes.io/projected/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kube-api-access-6k46x\") pod \"auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.191638 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.191605    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"]
Jul 07 02:38:52.194962 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:38:52.194935    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec05c28b_6542_423b_be5a_13387203eff7.slice/crio-47349c483511e2e3f1a0b18b0327132dc743aa6719c6cb4e6e1c2760151f57c0 WatchSource:0}: Error finding container 47349c483511e2e3f1a0b18b0327132dc743aa6719c6cb4e6e1c2760151f57c0: Status 404 returned error can't find the container with id 47349c483511e2e3f1a0b18b0327132dc743aa6719c6cb4e6e1c2760151f57c0
Jul 07 02:38:52.284809 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.284779    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:52.425940 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:52.425910    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"]
Jul 07 02:38:52.427932 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:38:52.427906    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e9aeb8_5913_4ab0_bbd4_bfd312698aa7.slice/crio-98934c5a422d84b796bac547d1a15d12940018da4ae811e372b135e5c8a0309b WatchSource:0}: Error finding container 98934c5a422d84b796bac547d1a15d12940018da4ae811e372b135e5c8a0309b: Status 404 returned error can't find the container with id 98934c5a422d84b796bac547d1a15d12940018da4ae811e372b135e5c8a0309b
Jul 07 02:38:53.037444 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:53.037401    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz" event={"ID":"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7","Type":"ContainerStarted","Data":"ea62d78d7540685092e535ed51ab1cb775105a3d64d3ad92f9cab79423ed1903"}
Jul 07 02:38:53.037888 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:53.037461    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz" event={"ID":"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7","Type":"ContainerStarted","Data":"98934c5a422d84b796bac547d1a15d12940018da4ae811e372b135e5c8a0309b"}
Jul 07 02:38:53.038964 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:53.038937    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" event={"ID":"ec05c28b-6542-423b-be5a-13387203eff7","Type":"ContainerStarted","Data":"89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4"}
Jul 07 02:38:53.039040 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:53.038982    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" event={"ID":"ec05c28b-6542-423b-be5a-13387203eff7","Type":"ContainerStarted","Data":"47349c483511e2e3f1a0b18b0327132dc743aa6719c6cb4e6e1c2760151f57c0"}
Jul 07 02:38:54.044750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:54.044719    2543 generic.go:356] "Generic (PLEG): container finished" podID="29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" containerID="ea62d78d7540685092e535ed51ab1cb775105a3d64d3ad92f9cab79423ed1903" exitCode=0
Jul 07 02:38:54.045220 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:54.044803    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz" event={"ID":"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7","Type":"ContainerDied","Data":"ea62d78d7540685092e535ed51ab1cb775105a3d64d3ad92f9cab79423ed1903"}
Jul 07 02:38:55.054331 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:55.054242    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz" event={"ID":"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7","Type":"ContainerStarted","Data":"ee262e2c633c7144b8ac7a8314644bb491dbcf5f5d4555b366988818337e6a4e"}
Jul 07 02:38:55.054331 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:55.054329    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:38:55.077090 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:55.077029    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz" podStartSLOduration=3.38029409 podStartE2EDuration="4.077007302s" totalImagesPullingTime="0s" totalInitContainerRuntime="696.713212ms" isStatefulPod=false podCreationTimestamp="2026-07-07 02:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:38:55.073916394 +0000 UTC m=+1247.717076917" watchObservedRunningTime="2026-07-07 02:38:55.077007302 +0000 UTC m=+1247.720167827"
Jul 07 02:38:58.069044 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:58.068964    2543 generic.go:356] "Generic (PLEG): container finished" podID="ec05c28b-6542-423b-be5a-13387203eff7" containerID="89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4" exitCode=0
Jul 07 02:38:58.069536 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:58.069048    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" event={"ID":"ec05c28b-6542-423b-be5a-13387203eff7","Type":"ContainerDied","Data":"89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4"}
Jul 07 02:38:59.076456 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:59.076410    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" event={"ID":"ec05c28b-6542-423b-be5a-13387203eff7","Type":"ContainerStarted","Data":"5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606"}
Jul 07 02:38:59.100818 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:38:59.100754    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podStartSLOduration=3.0009941 podStartE2EDuration="8.100731549s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.099737449s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:38:59.099012155 +0000 UTC m=+1251.742172677" watchObservedRunningTime="2026-07-07 02:38:59.100731549 +0000 UTC m=+1251.743892069"
Jul 07 02:39:00.046716 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:00.046686    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:39:02.046925 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:02.046886    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:39:02.048807 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:02.048780    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:39:12.047726 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:12.047683    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:39:22.046867 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:22.046825    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:39:26.060916 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:26.060879    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:39:27.694836 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:27.694803    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"]
Jul 07 02:39:32.047382 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:32.047335    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:39:42.047179 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:42.047135    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:39:42.716876 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:42.716826    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d" podUID="3a29a134-cf45-4008-a645-f15a6a0a0f22" containerName="main" containerID="cri-o://f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf" gracePeriod=45
Jul 07 02:39:42.983895 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:42.983824    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:39:43.076313 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076277    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xqfd\" (UniqueName: \"kubernetes.io/projected/3a29a134-cf45-4008-a645-f15a6a0a0f22-kube-api-access-4xqfd\") pod \"3a29a134-cf45-4008-a645-f15a6a0a0f22\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") "
Jul 07 02:39:43.076780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076358    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-tmp-dir\") pod \"3a29a134-cf45-4008-a645-f15a6a0a0f22\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") "
Jul 07 02:39:43.076780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076378    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-dshm\") pod \"3a29a134-cf45-4008-a645-f15a6a0a0f22\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") "
Jul 07 02:39:43.076780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076402    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a29a134-cf45-4008-a645-f15a6a0a0f22-tls-certs\") pod \"3a29a134-cf45-4008-a645-f15a6a0a0f22\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") "
Jul 07 02:39:43.076780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076432    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-home\") pod \"3a29a134-cf45-4008-a645-f15a6a0a0f22\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") "
Jul 07 02:39:43.076780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076509    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-model-cache\") pod \"3a29a134-cf45-4008-a645-f15a6a0a0f22\" (UID: \"3a29a134-cf45-4008-a645-f15a6a0a0f22\") "
Jul 07 02:39:43.076780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076714    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-home" (OuterVolumeSpecName: "home") pod "3a29a134-cf45-4008-a645-f15a6a0a0f22" (UID: "3a29a134-cf45-4008-a645-f15a6a0a0f22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:39:43.076780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076726    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a29a134-cf45-4008-a645-f15a6a0a0f22" (UID: "3a29a134-cf45-4008-a645-f15a6a0a0f22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:39:43.077155 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076863    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a29a134-cf45-4008-a645-f15a6a0a0f22" (UID: "3a29a134-cf45-4008-a645-f15a6a0a0f22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:39:43.077155 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076888    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:39:43.077155 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.076908    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:39:43.078834 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.078809    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a29a134-cf45-4008-a645-f15a6a0a0f22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a29a134-cf45-4008-a645-f15a6a0a0f22" (UID: "3a29a134-cf45-4008-a645-f15a6a0a0f22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:39:43.078960 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.078908    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a29a134-cf45-4008-a645-f15a6a0a0f22-kube-api-access-4xqfd" (OuterVolumeSpecName: "kube-api-access-4xqfd") pod "3a29a134-cf45-4008-a645-f15a6a0a0f22" (UID: "3a29a134-cf45-4008-a645-f15a6a0a0f22"). InnerVolumeSpecName "kube-api-access-4xqfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:39:43.078960 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.078927    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-dshm" (OuterVolumeSpecName: "dshm") pod "3a29a134-cf45-4008-a645-f15a6a0a0f22" (UID: "3a29a134-cf45-4008-a645-f15a6a0a0f22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:39:43.177684 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.177647    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:39:43.177684 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.177677    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xqfd\" (UniqueName: \"kubernetes.io/projected/3a29a134-cf45-4008-a645-f15a6a0a0f22-kube-api-access-4xqfd\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:39:43.177684 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.177688    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a29a134-cf45-4008-a645-f15a6a0a0f22-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:39:43.177684 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.177697    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a29a134-cf45-4008-a645-f15a6a0a0f22-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:39:43.303139 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.303047    2543 generic.go:356] "Generic (PLEG): container finished" podID="3a29a134-cf45-4008-a645-f15a6a0a0f22" containerID="f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf" exitCode=0
Jul 07 02:39:43.303139 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.303098    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"
Jul 07 02:39:43.303139 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.303106    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d" event={"ID":"3a29a134-cf45-4008-a645-f15a6a0a0f22","Type":"ContainerDied","Data":"f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf"}
Jul 07 02:39:43.303460 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.303149    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d" event={"ID":"3a29a134-cf45-4008-a645-f15a6a0a0f22","Type":"ContainerDied","Data":"dd621f9667d560440790d06285b8c1717fcb8921330b95f52dd2f44a60578c69"}
Jul 07 02:39:43.303460 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.303170    2543 scope.go:117] "RemoveContainer" containerID="f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf"
Jul 07 02:39:43.312451 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.312427    2543 scope.go:117] "RemoveContainer" containerID="f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf"
Jul 07 02:39:43.312732 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:39:43.312710    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf\": container with ID starting with f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf not found: ID does not exist" containerID="f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf"
Jul 07 02:39:43.312794 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.312741    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf"} err="failed to get container status \"f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf\": rpc error: code = NotFound desc = could not find container \"f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf\": container with ID starting with f80a33f73045528511562a7038cbe056304935d558171fd0186050d644b65dcf not found: ID does not exist"
Jul 07 02:39:43.324877 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.324851    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"]
Jul 07 02:39:43.328958 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.328930    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5d8ffd58ddmqw6d"]
Jul 07 02:39:43.979858 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:43.979815    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a29a134-cf45-4008-a645-f15a6a0a0f22" path="/var/lib/kubelet/pods/3a29a134-cf45-4008-a645-f15a6a0a0f22/volumes"
Jul 07 02:39:52.047130 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:39:52.047070    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:40:02.047617 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:02.047570    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:40:12.047289 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:12.047248    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:40:22.047203 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:22.047138    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:40:32.047355 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:32.047243    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:40:42.047899 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:42.047848    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:40:52.057260 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:52.057227    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:40:52.065108 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:52.065082    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:40:53.474161 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.474122    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"]
Jul 07 02:40:53.482926 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.482895    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"]
Jul 07 02:40:53.493618 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:40:53.493582    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:40:53.493618 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:40:53.493618 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 02:40:53.493618 ip-10-0-132-101 kubenswrapper[2543]:  > containerID="ee262e2c633c7144b8ac7a8314644bb491dbcf5f5d4555b366988818337e6a4e" cmd=["/bin/sleep","15"]
Jul 07 02:40:53.493934 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.493632    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz" message="" err=<
Jul 07 02:40:53.493934 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:40:53.493934 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 02:40:53.493934 ip-10-0-132-101 kubenswrapper[2543]:  >
Jul 07 02:40:53.493934 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:40:53.493646    2543 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:40:53.493934 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:40:53.493934 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 02:40:53.493934 ip-10-0-132-101 kubenswrapper[2543]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz" podUID="29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" containerName="main" containerID="cri-o://ee262e2c633c7144b8ac7a8314644bb491dbcf5f5d4555b366988818337e6a4e"
Jul 07 02:40:53.493934 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.493683    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz" podUID="29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" containerName="main" containerID="cri-o://ee262e2c633c7144b8ac7a8314644bb491dbcf5f5d4555b366988818337e6a4e" gracePeriod=60
Jul 07 02:40:53.652655 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.652624    2543 generic.go:356] "Generic (PLEG): container finished" podID="29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" containerID="ee262e2c633c7144b8ac7a8314644bb491dbcf5f5d4555b366988818337e6a4e" exitCode=0
Jul 07 02:40:53.652825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.652700    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz" event={"ID":"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7","Type":"ContainerDied","Data":"ee262e2c633c7144b8ac7a8314644bb491dbcf5f5d4555b366988818337e6a4e"}
Jul 07 02:40:53.749940 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.749918    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:40:53.914431 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.914398    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tls-certs\") pod \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") "
Jul 07 02:40:53.914637 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.914452    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6k46x\" (UniqueName: \"kubernetes.io/projected/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kube-api-access-6k46x\") pod \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") "
Jul 07 02:40:53.914637 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.914475    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kserve-provision-location\") pod \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") "
Jul 07 02:40:53.914637 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.914516    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tokenizer-uds\") pod \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\" (UID: \"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7\") "
Jul 07 02:40:53.914840 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.914813    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" (UID: "29e9aeb8-5913-4ab0-bbd4-bfd312698aa7"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:40:53.915267 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.915230    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" (UID: "29e9aeb8-5913-4ab0-bbd4-bfd312698aa7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:40:53.916768 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.916744    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" (UID: "29e9aeb8-5913-4ab0-bbd4-bfd312698aa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:40:53.916941 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:53.916920    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kube-api-access-6k46x" (OuterVolumeSpecName: "kube-api-access-6k46x") pod "29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" (UID: "29e9aeb8-5913-4ab0-bbd4-bfd312698aa7"). InnerVolumeSpecName "kube-api-access-6k46x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:40:54.015760 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:54.015726    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:40:54.015760 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:54.015758    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k46x\" (UniqueName: \"kubernetes.io/projected/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kube-api-access-6k46x\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:40:54.015963 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:54.015775    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:40:54.015963 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:54.015787    2543 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7-tokenizer-uds\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:40:54.659213 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:54.659168    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"
Jul 07 02:40:54.659641 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:54.659215    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz" event={"ID":"29e9aeb8-5913-4ab0-bbd4-bfd312698aa7","Type":"ContainerDied","Data":"98934c5a422d84b796bac547d1a15d12940018da4ae811e372b135e5c8a0309b"}
Jul 07 02:40:54.659641 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:54.659281    2543 scope.go:117] "RemoveContainer" containerID="ee262e2c633c7144b8ac7a8314644bb491dbcf5f5d4555b366988818337e6a4e"
Jul 07 02:40:54.667938 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:54.667917    2543 scope.go:117] "RemoveContainer" containerID="ea62d78d7540685092e535ed51ab1cb775105a3d64d3ad92f9cab79423ed1903"
Jul 07 02:40:54.677994 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:54.677973    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"]
Jul 07 02:40:54.683926 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:54.683903    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-c9c75d5f5-9gfqz"]
Jul 07 02:40:55.979840 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:40:55.979804    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" path="/var/lib/kubelet/pods/29e9aeb8-5913-4ab0-bbd4-bfd312698aa7/volumes"
Jul 07 02:41:02.758348 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.758317    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"]
Jul 07 02:41:02.758721 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.758630    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" containerName="main"
Jul 07 02:41:02.758721 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.758640    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" containerName="main"
Jul 07 02:41:02.758721 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.758652    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" containerName="storage-initializer"
Jul 07 02:41:02.758721 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.758657    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" containerName="storage-initializer"
Jul 07 02:41:02.758721 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.758672    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a29a134-cf45-4008-a645-f15a6a0a0f22" containerName="main"
Jul 07 02:41:02.758721 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.758677    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a29a134-cf45-4008-a645-f15a6a0a0f22" containerName="main"
Jul 07 02:41:02.758902 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.758732    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a29a134-cf45-4008-a645-f15a6a0a0f22" containerName="main"
Jul 07 02:41:02.758902 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.758740    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="29e9aeb8-5913-4ab0-bbd4-bfd312698aa7" containerName="main"
Jul 07 02:41:02.761691 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.761668    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:02.765317 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.765293    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-hrlld\""
Jul 07 02:41:02.765454 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.765315    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 07 02:41:02.775043 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.775016    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"]
Jul 07 02:41:02.893433 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.893389    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rjgz\" (UniqueName: \"kubernetes.io/projected/907eb484-984c-4559-8253-de1598451c63-kube-api-access-2rjgz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:02.893614 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.893474    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/907eb484-984c-4559-8253-de1598451c63-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:02.893614 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.893524    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907eb484-984c-4559-8253-de1598451c63-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:02.994908 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.994869    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/907eb484-984c-4559-8253-de1598451c63-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:02.995082 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.994929    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907eb484-984c-4559-8253-de1598451c63-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:02.995082 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.994987    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rjgz\" (UniqueName: \"kubernetes.io/projected/907eb484-984c-4559-8253-de1598451c63-kube-api-access-2rjgz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:02.995374 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.995353    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/907eb484-984c-4559-8253-de1598451c63-tokenizer-uds\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:02.997566 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:02.997546    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907eb484-984c-4559-8253-de1598451c63-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:03.004366 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:03.004343    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rjgz\" (UniqueName: \"kubernetes.io/projected/907eb484-984c-4559-8253-de1598451c63-kube-api-access-2rjgz\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") " pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:03.071473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:03.071380    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:03.212801 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:03.212719    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"]
Jul 07 02:41:03.216426 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:41:03.216395    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907eb484_984c_4559_8253_de1598451c63.slice/crio-0d9cefbbd8d82eafd8f25a1173e6ead627fb21ce3db2dc746646d9e8b615dfe6 WatchSource:0}: Error finding container 0d9cefbbd8d82eafd8f25a1173e6ead627fb21ce3db2dc746646d9e8b615dfe6: Status 404 returned error can't find the container with id 0d9cefbbd8d82eafd8f25a1173e6ead627fb21ce3db2dc746646d9e8b615dfe6
Jul 07 02:41:03.701683 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:03.701651    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8" event={"ID":"907eb484-984c-4559-8253-de1598451c63","Type":"ContainerStarted","Data":"8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e"}
Jul 07 02:41:03.701859 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:03.701700    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8" event={"ID":"907eb484-984c-4559-8253-de1598451c63","Type":"ContainerStarted","Data":"0d9cefbbd8d82eafd8f25a1173e6ead627fb21ce3db2dc746646d9e8b615dfe6"}
Jul 07 02:41:03.701859 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:03.701851    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:03.719851 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:03.719798    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8" podStartSLOduration=1.71978174 podStartE2EDuration="1.71978174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:41:03.71899463 +0000 UTC m=+1376.362155148" watchObservedRunningTime="2026-07-07 02:41:03.71978174 +0000 UTC m=+1376.362942262"
Jul 07 02:41:08.677277 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:08.677203    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" containerID="cri-o://5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606" gracePeriod=45
Jul 07 02:41:09.058466 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.058368    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": dial tcp 10.134.0.37:8000: connect: connection refused"
Jul 07 02:41:09.429590 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.429568    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:41:09.551958 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.551921    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec05c28b-6542-423b-be5a-13387203eff7-tls-certs\") pod \"ec05c28b-6542-423b-be5a-13387203eff7\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") "
Jul 07 02:41:09.552148 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.551970    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-kserve-provision-location\") pod \"ec05c28b-6542-423b-be5a-13387203eff7\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") "
Jul 07 02:41:09.552148 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.552006    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrrls\" (UniqueName: \"kubernetes.io/projected/ec05c28b-6542-423b-be5a-13387203eff7-kube-api-access-qrrls\") pod \"ec05c28b-6542-423b-be5a-13387203eff7\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") "
Jul 07 02:41:09.552148 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.552056    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-dshm\") pod \"ec05c28b-6542-423b-be5a-13387203eff7\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") "
Jul 07 02:41:09.552148 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.552121    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-tmp-dir\") pod \"ec05c28b-6542-423b-be5a-13387203eff7\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") "
Jul 07 02:41:09.552148 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.552148    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-home\") pod \"ec05c28b-6542-423b-be5a-13387203eff7\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") "
Jul 07 02:41:09.552448 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.552212    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-model-cache\") pod \"ec05c28b-6542-423b-be5a-13387203eff7\" (UID: \"ec05c28b-6542-423b-be5a-13387203eff7\") "
Jul 07 02:41:09.552556 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.552525    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-model-cache" (OuterVolumeSpecName: "model-cache") pod "ec05c28b-6542-423b-be5a-13387203eff7" (UID: "ec05c28b-6542-423b-be5a-13387203eff7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:41:09.552940 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.552909    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-home" (OuterVolumeSpecName: "home") pod "ec05c28b-6542-423b-be5a-13387203eff7" (UID: "ec05c28b-6542-423b-be5a-13387203eff7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:41:09.554398 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.554370    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec05c28b-6542-423b-be5a-13387203eff7-kube-api-access-qrrls" (OuterVolumeSpecName: "kube-api-access-qrrls") pod "ec05c28b-6542-423b-be5a-13387203eff7" (UID: "ec05c28b-6542-423b-be5a-13387203eff7"). InnerVolumeSpecName "kube-api-access-qrrls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:41:09.554862 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.554842    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-dshm" (OuterVolumeSpecName: "dshm") pod "ec05c28b-6542-423b-be5a-13387203eff7" (UID: "ec05c28b-6542-423b-be5a-13387203eff7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:41:09.554926 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.554866    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec05c28b-6542-423b-be5a-13387203eff7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec05c28b-6542-423b-be5a-13387203eff7" (UID: "ec05c28b-6542-423b-be5a-13387203eff7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:41:09.563866 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.563835    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ec05c28b-6542-423b-be5a-13387203eff7" (UID: "ec05c28b-6542-423b-be5a-13387203eff7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:41:09.619152 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.619098    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec05c28b-6542-423b-be5a-13387203eff7" (UID: "ec05c28b-6542-423b-be5a-13387203eff7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:41:09.653047 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.653019    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:41:09.653047 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.653045    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:41:09.653207 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.653056    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:41:09.653207 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.653066    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:41:09.653207 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.653073    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec05c28b-6542-423b-be5a-13387203eff7-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:41:09.653207 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.653082    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec05c28b-6542-423b-be5a-13387203eff7-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:41:09.653207 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.653092    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrls\" (UniqueName: \"kubernetes.io/projected/ec05c28b-6542-423b-be5a-13387203eff7-kube-api-access-qrrls\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:41:09.733471 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.733433    2543 generic.go:356] "Generic (PLEG): container finished" podID="ec05c28b-6542-423b-be5a-13387203eff7" containerID="5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606" exitCode=0
Jul 07 02:41:09.733850 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.733482    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"
Jul 07 02:41:09.733850 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.733484    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" event={"ID":"ec05c28b-6542-423b-be5a-13387203eff7","Type":"ContainerDied","Data":"5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606"}
Jul 07 02:41:09.733850 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.733618    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b" event={"ID":"ec05c28b-6542-423b-be5a-13387203eff7","Type":"ContainerDied","Data":"47349c483511e2e3f1a0b18b0327132dc743aa6719c6cb4e6e1c2760151f57c0"}
Jul 07 02:41:09.733850 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.733640    2543 scope.go:117] "RemoveContainer" containerID="5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606"
Jul 07 02:41:09.742790 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.742771    2543 scope.go:117] "RemoveContainer" containerID="89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4"
Jul 07 02:41:09.750901 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.750880    2543 scope.go:117] "RemoveContainer" containerID="5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606"
Jul 07 02:41:09.751167 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:41:09.751148    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606\": container with ID starting with 5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606 not found: ID does not exist" containerID="5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606"
Jul 07 02:41:09.751358 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.751180    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606"} err="failed to get container status \"5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606\": rpc error: code = NotFound desc = could not find container \"5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606\": container with ID starting with 5709b4217d69903b2bd535a59132c78bb9fe8b34f556ad917ac7056ca1b9a606 not found: ID does not exist"
Jul 07 02:41:09.751358 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.751231    2543 scope.go:117] "RemoveContainer" containerID="89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4"
Jul 07 02:41:09.751484 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:41:09.751465    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4\": container with ID starting with 89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4 not found: ID does not exist" containerID="89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4"
Jul 07 02:41:09.751529 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.751493    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4"} err="failed to get container status \"89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4\": rpc error: code = NotFound desc = could not find container \"89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4\": container with ID starting with 89eb29e45413580c9e4d4afd663473c32031893793de7c226ef5291e0b1e5bd4 not found: ID does not exist"
Jul 07 02:41:09.756726 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.756705    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"]
Jul 07 02:41:09.761625 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.761602    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-666bbbb8df-6gg4b"]
Jul 07 02:41:09.979916 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:09.979834    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec05c28b-6542-423b-be5a-13387203eff7" path="/var/lib/kubelet/pods/ec05c28b-6542-423b-be5a-13387203eff7/volumes"
Jul 07 02:41:29.451037 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.450996    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"]
Jul 07 02:41:29.451654 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.451521    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main"
Jul 07 02:41:29.451654 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.451543    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main"
Jul 07 02:41:29.451654 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.451565    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="storage-initializer"
Jul 07 02:41:29.451654 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.451575    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="storage-initializer"
Jul 07 02:41:29.451654 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.451651    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec05c28b-6542-423b-be5a-13387203eff7" containerName="main"
Jul 07 02:41:29.455041 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.455017    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.458481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.458460    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 07 02:41:29.458712 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.458698    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:41:29.469642 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.469617    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"]
Jul 07 02:41:29.512422 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.512387    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-dshm\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.512422 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.512420    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-model-cache\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.512627 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.512440    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a17368-b192-413e-a59c-9a0705c9ab60-tls-certs\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.512627 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.512456    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-tmp-dir\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.512627 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.512534    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-kserve-provision-location\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.512627 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.512599    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59gbc\" (UniqueName: \"kubernetes.io/projected/f3a17368-b192-413e-a59c-9a0705c9ab60-kube-api-access-59gbc\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.512752 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.512661    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-home\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.613924 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.613889    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59gbc\" (UniqueName: \"kubernetes.io/projected/f3a17368-b192-413e-a59c-9a0705c9ab60-kube-api-access-59gbc\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.614122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.613941    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-home\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.614122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.613973    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-dshm\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.614122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.613989    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-model-cache\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.614122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.614006    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a17368-b192-413e-a59c-9a0705c9ab60-tls-certs\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.614122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.614103    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-tmp-dir\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.614442 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.614156    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-kserve-provision-location\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.614529 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.614508    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-home\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.614588 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.614559    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-model-cache\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.614720 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.614588    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-tmp-dir\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.614720 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.614616    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-kserve-provision-location\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.616928 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.616905    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-dshm\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.617146 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.617129    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a17368-b192-413e-a59c-9a0705c9ab60-tls-certs\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.622990 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.622971    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59gbc\" (UniqueName: \"kubernetes.io/projected/f3a17368-b192-413e-a59c-9a0705c9ab60-kube-api-access-59gbc\") pod \"auth-disabled-test-kserve-5596879d7-gch2p\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.767363 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.767275    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:29.918548 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:29.918525    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"]
Jul 07 02:41:29.920548 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:41:29.920515    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a17368_b192_413e_a59c_9a0705c9ab60.slice/crio-68782f2a02950d019371012c8ed358a426eb8aa9b7a04177843c224b371340a5 WatchSource:0}: Error finding container 68782f2a02950d019371012c8ed358a426eb8aa9b7a04177843c224b371340a5: Status 404 returned error can't find the container with id 68782f2a02950d019371012c8ed358a426eb8aa9b7a04177843c224b371340a5
Jul 07 02:41:30.844040 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:30.843991    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" event={"ID":"f3a17368-b192-413e-a59c-9a0705c9ab60","Type":"ContainerStarted","Data":"7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc"}
Jul 07 02:41:30.844514 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:30.844061    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" event={"ID":"f3a17368-b192-413e-a59c-9a0705c9ab60","Type":"ContainerStarted","Data":"68782f2a02950d019371012c8ed358a426eb8aa9b7a04177843c224b371340a5"}
Jul 07 02:41:34.709974 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:34.709946    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:34.864665 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:34.864633    2543 generic.go:356] "Generic (PLEG): container finished" podID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerID="7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc" exitCode=0
Jul 07 02:41:34.864837 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:34.864711    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" event={"ID":"f3a17368-b192-413e-a59c-9a0705c9ab60","Type":"ContainerDied","Data":"7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc"}
Jul 07 02:41:35.871578 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:35.871544    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" event={"ID":"f3a17368-b192-413e-a59c-9a0705c9ab60","Type":"ContainerStarted","Data":"577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e"}
Jul 07 02:41:35.896668 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:35.896597    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podStartSLOduration=2.862629416 podStartE2EDuration="6.896578751s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.033949335s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:41:35.894428604 +0000 UTC m=+1408.537589126" watchObservedRunningTime="2026-07-07 02:41:35.896578751 +0000 UTC m=+1408.539739273"
Jul 07 02:41:36.767569 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:36.767531    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:39.768004 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:39.767965    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:41:39.769637 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:39.769608    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:41:45.059915 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.059882    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"]
Jul 07 02:41:45.069361 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:41:45.069318    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 02:41:45.069361 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:41:45.069361 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 02:41:45.069361 ip-10-0-132-101 kubenswrapper[2543]:  > containerID="8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e" cmd=["/bin/sleep","15"]
Jul 07 02:41:45.069691 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.069381    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8" message="" err=<
Jul 07 02:41:45.069691 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:41:45.069691 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 02:41:45.069691 ip-10-0-132-101 kubenswrapper[2543]:  >
Jul 07 02:41:45.069691 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:41:45.069398    2543 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 02:41:45.069691 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 02:41:45.069691 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 02:41:45.069691 ip-10-0-132-101 kubenswrapper[2543]:  > pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8" podUID="907eb484-984c-4559-8253-de1598451c63" containerName="main" containerID="cri-o://8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e"
Jul 07 02:41:45.069691 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.069448    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8" podUID="907eb484-984c-4559-8253-de1598451c63" containerName="main" containerID="cri-o://8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e" gracePeriod=60
Jul 07 02:41:45.325980 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.325942    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:45.462078 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.462038    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907eb484-984c-4559-8253-de1598451c63-tls-certs\") pod \"907eb484-984c-4559-8253-de1598451c63\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") "
Jul 07 02:41:45.462313 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.462105    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/907eb484-984c-4559-8253-de1598451c63-tokenizer-uds\") pod \"907eb484-984c-4559-8253-de1598451c63\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") "
Jul 07 02:41:45.462313 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.462258    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rjgz\" (UniqueName: \"kubernetes.io/projected/907eb484-984c-4559-8253-de1598451c63-kube-api-access-2rjgz\") pod \"907eb484-984c-4559-8253-de1598451c63\" (UID: \"907eb484-984c-4559-8253-de1598451c63\") "
Jul 07 02:41:45.462429 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.462404    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/907eb484-984c-4559-8253-de1598451c63-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "907eb484-984c-4559-8253-de1598451c63" (UID: "907eb484-984c-4559-8253-de1598451c63"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:41:45.462544 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.462528    2543 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/907eb484-984c-4559-8253-de1598451c63-tokenizer-uds\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:41:45.464362 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.464333    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907eb484-984c-4559-8253-de1598451c63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "907eb484-984c-4559-8253-de1598451c63" (UID: "907eb484-984c-4559-8253-de1598451c63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:41:45.464569 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.464545    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907eb484-984c-4559-8253-de1598451c63-kube-api-access-2rjgz" (OuterVolumeSpecName: "kube-api-access-2rjgz") pod "907eb484-984c-4559-8253-de1598451c63" (UID: "907eb484-984c-4559-8253-de1598451c63"). InnerVolumeSpecName "kube-api-access-2rjgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:41:45.563073 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.562980    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rjgz\" (UniqueName: \"kubernetes.io/projected/907eb484-984c-4559-8253-de1598451c63-kube-api-access-2rjgz\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:41:45.563073 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.563008    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/907eb484-984c-4559-8253-de1598451c63-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:41:45.921840 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.921809    2543 generic.go:356] "Generic (PLEG): container finished" podID="907eb484-984c-4559-8253-de1598451c63" containerID="8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e" exitCode=0
Jul 07 02:41:45.922016 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.921864    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"
Jul 07 02:41:45.922016 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.921900    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8" event={"ID":"907eb484-984c-4559-8253-de1598451c63","Type":"ContainerDied","Data":"8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e"}
Jul 07 02:41:45.922016 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.921953    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8" event={"ID":"907eb484-984c-4559-8253-de1598451c63","Type":"ContainerDied","Data":"0d9cefbbd8d82eafd8f25a1173e6ead627fb21ce3db2dc746646d9e8b615dfe6"}
Jul 07 02:41:45.922016 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.921974    2543 scope.go:117] "RemoveContainer" containerID="8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e"
Jul 07 02:41:45.931941 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.931922    2543 scope.go:117] "RemoveContainer" containerID="8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e"
Jul 07 02:41:45.932250 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:41:45.932223    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e\": container with ID starting with 8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e not found: ID does not exist" containerID="8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e"
Jul 07 02:41:45.932344 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.932259    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e"} err="failed to get container status \"8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e\": rpc error: code = NotFound desc = could not find container \"8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e\": container with ID starting with 8bbba6b6d347b1d553231975065d207f0f4d5276688884cc97dc8aeaae8d4c1e not found: ID does not exist"
Jul 07 02:41:45.941973 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.941949    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"]
Jul 07 02:41:45.946311 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.946289    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-scheqsdc8"]
Jul 07 02:41:45.980964 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:45.980930    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="907eb484-984c-4559-8253-de1598451c63" path="/var/lib/kubelet/pods/907eb484-984c-4559-8253-de1598451c63/volumes"
Jul 07 02:41:49.768211 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:49.768141    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:41:59.768364 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:41:59.768264    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:42:09.768043 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:09.767995    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:42:19.768540 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:19.768496    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:42:29.768152 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:29.768107    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:42:32.939260 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:32.939227    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"]
Jul 07 02:42:32.939791 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:32.939769    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="907eb484-984c-4559-8253-de1598451c63" containerName="main"
Jul 07 02:42:32.939791 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:32.939792    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="907eb484-984c-4559-8253-de1598451c63" containerName="main"
Jul 07 02:42:32.939920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:32.939893    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="907eb484-984c-4559-8253-de1598451c63" containerName="main"
Jul 07 02:42:32.943445 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:32.943426    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:32.946324 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:32.946297    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 02:42:32.946445 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:32.946297    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 07 02:42:32.954280 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:32.954249    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"]
Jul 07 02:42:33.064963 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.064919    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.064963 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.064962    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.065181 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.065001    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.065181 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.065066    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2qrp\" (UniqueName: \"kubernetes.io/projected/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kube-api-access-s2qrp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.065181 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.065113    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.065181 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.065150    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caa3a2b9-414f-471b-82ec-75ae639d6ce4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.065367 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.065221    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.065367 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.065252    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.166487 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.166452    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.166688 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.166501    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.166688 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.166517    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.166688 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.166546    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.166688 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.166563    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2qrp\" (UniqueName: \"kubernetes.io/projected/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kube-api-access-s2qrp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.166688 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.166582    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.166688 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.166604    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caa3a2b9-414f-471b-82ec-75ae639d6ce4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.166688 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.166653    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.167084 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.167056    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.167409 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.167378    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.167515 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.167144    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.167515 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.167089    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.167708 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.167687    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caa3a2b9-414f-471b-82ec-75ae639d6ce4-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.169035 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.169009    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.169177 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.169160    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.176251 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.176215    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2qrp\" (UniqueName: \"kubernetes.io/projected/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kube-api-access-s2qrp\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.254801 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.254697    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:33.407271 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:33.407226    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"]
Jul 07 02:42:33.408643 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:42:33.408614    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa3a2b9_414f_471b_82ec_75ae639d6ce4.slice/crio-8d5f247c5f9d147d93a840da6d5f39e403e0cdeea315fd4f7ab17f4c182af21a WatchSource:0}: Error finding container 8d5f247c5f9d147d93a840da6d5f39e403e0cdeea315fd4f7ab17f4c182af21a: Status 404 returned error can't find the container with id 8d5f247c5f9d147d93a840da6d5f39e403e0cdeea315fd4f7ab17f4c182af21a
Jul 07 02:42:34.154205 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:34.154151    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" event={"ID":"caa3a2b9-414f-471b-82ec-75ae639d6ce4","Type":"ContainerStarted","Data":"1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81"}
Jul 07 02:42:34.154700 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:34.154244    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" event={"ID":"caa3a2b9-414f-471b-82ec-75ae639d6ce4","Type":"ContainerStarted","Data":"8d5f247c5f9d147d93a840da6d5f39e403e0cdeea315fd4f7ab17f4c182af21a"}
Jul 07 02:42:39.768636 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:39.768589    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:42:40.187116 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:40.187083    2543 generic.go:356] "Generic (PLEG): container finished" podID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerID="1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81" exitCode=0
Jul 07 02:42:40.187292 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:40.187138    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" event={"ID":"caa3a2b9-414f-471b-82ec-75ae639d6ce4","Type":"ContainerDied","Data":"1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81"}
Jul 07 02:42:41.201682 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:41.201641    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" event={"ID":"caa3a2b9-414f-471b-82ec-75ae639d6ce4","Type":"ContainerStarted","Data":"9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0"}
Jul 07 02:42:41.226638 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:41.226569    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podStartSLOduration=3.10701914 podStartE2EDuration="9.22654513s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.11952599s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:42:41.223024874 +0000 UTC m=+1473.866185417" watchObservedRunningTime="2026-07-07 02:42:41.22654513 +0000 UTC m=+1473.869705654"
Jul 07 02:42:41.255505 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:41.255467    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:43.255055 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:43.255015    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:42:43.256852 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:43.256822    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 07 02:42:49.768041 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:49.767988    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:42:53.256029 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:53.255974    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 07 02:42:59.768652 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:42:59.768612    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:43:03.255436 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:03.255387    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 07 02:43:07.918725 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:07.918687    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:43:07.921795 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:07.921763    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:43:09.768310 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:09.768265    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:43:13.255652 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:13.255605    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 07 02:43:19.768556 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:19.768502    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:43:23.255525 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:23.255468    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 07 02:43:29.768710 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:29.768592    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:43:33.255402 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:33.255353    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 07 02:43:39.778403 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:39.778369    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:43:39.787030 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:39.787004    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:43:43.255515 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:43.255461    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 07 02:43:45.334716 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:45.334681    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"]
Jul 07 02:43:53.255662 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:43:53.255612    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": dial tcp 10.134.0.41:8000: connect: connection refused"
Jul 07 02:44:00.365664 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:00.365620    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" containerID="cri-o://577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e" gracePeriod=45
Jul 07 02:44:00.780124 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:00.780021    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": dial tcp 10.134.0.40:8000: connect: connection refused"
Jul 07 02:44:01.134961 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.134929    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:44:01.263055 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.263025    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-kserve-provision-location\") pod \"f3a17368-b192-413e-a59c-9a0705c9ab60\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") "
Jul 07 02:44:01.263263 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.263068    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-model-cache\") pod \"f3a17368-b192-413e-a59c-9a0705c9ab60\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") "
Jul 07 02:44:01.263263 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.263109    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-home\") pod \"f3a17368-b192-413e-a59c-9a0705c9ab60\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") "
Jul 07 02:44:01.263263 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.263176    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-tmp-dir\") pod \"f3a17368-b192-413e-a59c-9a0705c9ab60\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") "
Jul 07 02:44:01.263426 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.263288    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-dshm\") pod \"f3a17368-b192-413e-a59c-9a0705c9ab60\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") "
Jul 07 02:44:01.263426 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.263315    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59gbc\" (UniqueName: \"kubernetes.io/projected/f3a17368-b192-413e-a59c-9a0705c9ab60-kube-api-access-59gbc\") pod \"f3a17368-b192-413e-a59c-9a0705c9ab60\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") "
Jul 07 02:44:01.263426 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.263346    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a17368-b192-413e-a59c-9a0705c9ab60-tls-certs\") pod \"f3a17368-b192-413e-a59c-9a0705c9ab60\" (UID: \"f3a17368-b192-413e-a59c-9a0705c9ab60\") "
Jul 07 02:44:01.263580 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.263425    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-model-cache" (OuterVolumeSpecName: "model-cache") pod "f3a17368-b192-413e-a59c-9a0705c9ab60" (UID: "f3a17368-b192-413e-a59c-9a0705c9ab60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:01.263720 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.263691    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:01.263842 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.263818    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-home" (OuterVolumeSpecName: "home") pod "f3a17368-b192-413e-a59c-9a0705c9ab60" (UID: "f3a17368-b192-413e-a59c-9a0705c9ab60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:01.265690 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.265666    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-dshm" (OuterVolumeSpecName: "dshm") pod "f3a17368-b192-413e-a59c-9a0705c9ab60" (UID: "f3a17368-b192-413e-a59c-9a0705c9ab60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:01.265987 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.265971    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a17368-b192-413e-a59c-9a0705c9ab60-kube-api-access-59gbc" (OuterVolumeSpecName: "kube-api-access-59gbc") pod "f3a17368-b192-413e-a59c-9a0705c9ab60" (UID: "f3a17368-b192-413e-a59c-9a0705c9ab60"). InnerVolumeSpecName "kube-api-access-59gbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:44:01.266051 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.266007    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3a17368-b192-413e-a59c-9a0705c9ab60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3a17368-b192-413e-a59c-9a0705c9ab60" (UID: "f3a17368-b192-413e-a59c-9a0705c9ab60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:44:01.276913 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.276886    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f3a17368-b192-413e-a59c-9a0705c9ab60" (UID: "f3a17368-b192-413e-a59c-9a0705c9ab60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:01.333661 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.333561    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f3a17368-b192-413e-a59c-9a0705c9ab60" (UID: "f3a17368-b192-413e-a59c-9a0705c9ab60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:01.364327 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.364292    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:01.364327 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.364322    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:01.364327 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.364333    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:01.364562 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.364342    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59gbc\" (UniqueName: \"kubernetes.io/projected/f3a17368-b192-413e-a59c-9a0705c9ab60-kube-api-access-59gbc\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:01.364562 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.364352    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3a17368-b192-413e-a59c-9a0705c9ab60-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:01.364562 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.364360    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f3a17368-b192-413e-a59c-9a0705c9ab60-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:01.628844 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.628809    2543 generic.go:356] "Generic (PLEG): container finished" podID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerID="577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e" exitCode=0
Jul 07 02:44:01.629265 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.628860    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" event={"ID":"f3a17368-b192-413e-a59c-9a0705c9ab60","Type":"ContainerDied","Data":"577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e"}
Jul 07 02:44:01.629265 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.628904    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p" event={"ID":"f3a17368-b192-413e-a59c-9a0705c9ab60","Type":"ContainerDied","Data":"68782f2a02950d019371012c8ed358a426eb8aa9b7a04177843c224b371340a5"}
Jul 07 02:44:01.629265 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.628926    2543 scope.go:117] "RemoveContainer" containerID="577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e"
Jul 07 02:44:01.629265 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.628863    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"
Jul 07 02:44:01.638370 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.638353    2543 scope.go:117] "RemoveContainer" containerID="7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc"
Jul 07 02:44:01.646567 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.646544    2543 scope.go:117] "RemoveContainer" containerID="577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e"
Jul 07 02:44:01.646821 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:44:01.646801    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e\": container with ID starting with 577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e not found: ID does not exist" containerID="577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e"
Jul 07 02:44:01.646866 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.646829    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e"} err="failed to get container status \"577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e\": rpc error: code = NotFound desc = could not find container \"577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e\": container with ID starting with 577f9d4c65279402e6cc641f5ecfdb6642549906ed7e94b7c60eb9277efe927e not found: ID does not exist"
Jul 07 02:44:01.646866 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.646845    2543 scope.go:117] "RemoveContainer" containerID="7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc"
Jul 07 02:44:01.647092 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:44:01.647073    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc\": container with ID starting with 7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc not found: ID does not exist" containerID="7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc"
Jul 07 02:44:01.647146 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.647099    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc"} err="failed to get container status \"7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc\": rpc error: code = NotFound desc = could not find container \"7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc\": container with ID starting with 7c47bdb3abc380987319e14af1d3fb948ab47ffe4a799b7332523ce5351795cc not found: ID does not exist"
Jul 07 02:44:01.652022 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.651998    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"]
Jul 07 02:44:01.656262 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.656239    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-5596879d7-gch2p"]
Jul 07 02:44:01.980368 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:01.980289    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" path="/var/lib/kubelet/pods/f3a17368-b192-413e-a59c-9a0705c9ab60/volumes"
Jul 07 02:44:03.265019 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:03.264990    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:44:03.272787 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:03.272762    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:44:27.974411 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:27.974353    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 07 02:44:29.602861 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:29.602827    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"]
Jul 07 02:44:30.774353 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:30.774319    2543 generic.go:356] "Generic (PLEG): container finished" podID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerID="465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b" exitCode=0
Jul 07 02:44:30.774765 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:30.774381    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" event={"ID":"61609f51-bd34-49c2-89b2-b7ee16ba35c4","Type":"ContainerDied","Data":"465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b"}
Jul 07 02:44:30.774765 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:30.774749    2543 scope.go:117] "RemoveContainer" containerID="465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b"
Jul 07 02:44:30.775706 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:30.775673    2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 07 02:44:30.775782 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:30.775767    2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 07 02:44:30.775824 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:30.775807    2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 07 02:44:31.780320 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:31.780289    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" event={"ID":"61609f51-bd34-49c2-89b2-b7ee16ba35c4","Type":"ContainerStarted","Data":"86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94"}
Jul 07 02:44:32.692748 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:32.692700    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:44:32.693005 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:32.692966    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 07 02:44:33.693170 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:33.693119    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 07 02:44:34.693067 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:34.693010    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 07 02:44:35.693543 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:35.693496    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 07 02:44:36.692955 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:36.692898    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Jul 07 02:44:37.696566 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:37.696534    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:44:37.697213 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:37.696838    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:44:37.697469 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:37.697451    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 02:44:44.635364 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:44.635293    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main" containerID="cri-o://9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0" gracePeriod=45
Jul 07 02:44:45.291056 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.291032    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:44:45.468824 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.468793    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kserve-provision-location\") pod \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") "
Jul 07 02:44:45.469038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.468855    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tls-certs\") pod \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") "
Jul 07 02:44:45.469038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.468898    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caa3a2b9-414f-471b-82ec-75ae639d6ce4-cabundle-cert\") pod \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") "
Jul 07 02:44:45.469038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.468921    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2qrp\" (UniqueName: \"kubernetes.io/projected/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kube-api-access-s2qrp\") pod \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") "
Jul 07 02:44:45.469038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.468952    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-model-cache\") pod \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") "
Jul 07 02:44:45.469038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.468985    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tmp-dir\") pod \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") "
Jul 07 02:44:45.469339 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.469053    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-home\") pod \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") "
Jul 07 02:44:45.469339 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.469092    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-dshm\") pod \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\" (UID: \"caa3a2b9-414f-471b-82ec-75ae639d6ce4\") "
Jul 07 02:44:45.469339 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.469302    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa3a2b9-414f-471b-82ec-75ae639d6ce4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "caa3a2b9-414f-471b-82ec-75ae639d6ce4" (UID: "caa3a2b9-414f-471b-82ec-75ae639d6ce4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:44:45.469499 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.469330    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-model-cache" (OuterVolumeSpecName: "model-cache") pod "caa3a2b9-414f-471b-82ec-75ae639d6ce4" (UID: "caa3a2b9-414f-471b-82ec-75ae639d6ce4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:45.469499 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.469408    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/caa3a2b9-414f-471b-82ec-75ae639d6ce4-cabundle-cert\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:45.469499 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.469428    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:45.469729 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.469704    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-home" (OuterVolumeSpecName: "home") pod "caa3a2b9-414f-471b-82ec-75ae639d6ce4" (UID: "caa3a2b9-414f-471b-82ec-75ae639d6ce4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:45.471715 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.471660    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-dshm" (OuterVolumeSpecName: "dshm") pod "caa3a2b9-414f-471b-82ec-75ae639d6ce4" (UID: "caa3a2b9-414f-471b-82ec-75ae639d6ce4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:45.471837 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.471727    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "caa3a2b9-414f-471b-82ec-75ae639d6ce4" (UID: "caa3a2b9-414f-471b-82ec-75ae639d6ce4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:44:45.471837 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.471740    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kube-api-access-s2qrp" (OuterVolumeSpecName: "kube-api-access-s2qrp") pod "caa3a2b9-414f-471b-82ec-75ae639d6ce4" (UID: "caa3a2b9-414f-471b-82ec-75ae639d6ce4"). InnerVolumeSpecName "kube-api-access-s2qrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:44:45.482262 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.482233    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "caa3a2b9-414f-471b-82ec-75ae639d6ce4" (UID: "caa3a2b9-414f-471b-82ec-75ae639d6ce4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:45.534500 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.534455    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "caa3a2b9-414f-471b-82ec-75ae639d6ce4" (UID: "caa3a2b9-414f-471b-82ec-75ae639d6ce4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:44:45.570840 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.570804    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:45.570840 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.570833    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:45.571028 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.570846    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:45.571028 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.570862    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:45.571028 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.570872    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2qrp\" (UniqueName: \"kubernetes.io/projected/caa3a2b9-414f-471b-82ec-75ae639d6ce4-kube-api-access-s2qrp\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:45.571028 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.570901    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caa3a2b9-414f-471b-82ec-75ae639d6ce4-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:44:45.851687 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.851591    2543 generic.go:356] "Generic (PLEG): container finished" podID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerID="9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0" exitCode=0
Jul 07 02:44:45.851687 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.851657    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"
Jul 07 02:44:45.852236 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.851692    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" event={"ID":"caa3a2b9-414f-471b-82ec-75ae639d6ce4","Type":"ContainerDied","Data":"9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0"}
Jul 07 02:44:45.852236 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.851751    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" event={"ID":"caa3a2b9-414f-471b-82ec-75ae639d6ce4","Type":"ContainerDied","Data":"8d5f247c5f9d147d93a840da6d5f39e403e0cdeea315fd4f7ab17f4c182af21a"}
Jul 07 02:44:45.852236 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.851774    2543 scope.go:117] "RemoveContainer" containerID="9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0"
Jul 07 02:44:45.861331 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.861311    2543 scope.go:117] "RemoveContainer" containerID="1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81"
Jul 07 02:44:45.869534 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.869516    2543 scope.go:117] "RemoveContainer" containerID="9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0"
Jul 07 02:44:45.869775 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:44:45.869758    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0\": container with ID starting with 9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0 not found: ID does not exist" containerID="9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0"
Jul 07 02:44:45.869820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.869786    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0"} err="failed to get container status \"9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0\": rpc error: code = NotFound desc = could not find container \"9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0\": container with ID starting with 9fb977abb6a66f656348e20aff0eb8b8ae51f03080c463507cfcf8e7fecee0b0 not found: ID does not exist"
Jul 07 02:44:45.869820 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.869810    2543 scope.go:117] "RemoveContainer" containerID="1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81"
Jul 07 02:44:45.870051 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:44:45.870031    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81\": container with ID starting with 1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81 not found: ID does not exist" containerID="1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81"
Jul 07 02:44:45.870106 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.870057    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81"} err="failed to get container status \"1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81\": rpc error: code = NotFound desc = could not find container \"1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81\": container with ID starting with 1c73019a4042e965983069ed564c0e208be6ba03dedb13e5faac54a472780c81 not found: ID does not exist"
Jul 07 02:44:45.875370 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.875344    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"]
Jul 07 02:44:45.880619 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.880595    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj"]
Jul 07 02:44:45.983771 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:45.983737    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" path="/var/lib/kubelet/pods/caa3a2b9-414f-471b-82ec-75ae639d6ce4/volumes"
Jul 07 02:44:46.266009 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:46.265945    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-c76db9948-pzbnj" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": context deadline exceeded"
Jul 07 02:44:53.930753 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.930723    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"]
Jul 07 02:44:53.931122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.931027    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="storage-initializer"
Jul 07 02:44:53.931122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.931037    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="storage-initializer"
Jul 07 02:44:53.931122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.931059    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main"
Jul 07 02:44:53.931122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.931064    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main"
Jul 07 02:44:53.931122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.931075    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main"
Jul 07 02:44:53.931122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.931081    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main"
Jul 07 02:44:53.931122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.931086    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="storage-initializer"
Jul 07 02:44:53.931122 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.931091    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="storage-initializer"
Jul 07 02:44:53.931451 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.931143    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a17368-b192-413e-a59c-9a0705c9ab60" containerName="main"
Jul 07 02:44:53.931451 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.931150    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="caa3a2b9-414f-471b-82ec-75ae639d6ce4" containerName="main"
Jul 07 02:44:53.934219 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.934181    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:53.937126 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.937102    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 02:44:53.937294 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.937137    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 07 02:44:53.937294 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.937104    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:44:53.937294 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.937220    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:44:53.945886 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:53.945863    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"]
Jul 07 02:44:54.046486 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.046454    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.046683 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.046490    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.046683 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.046528    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.046683 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.046606    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.046683 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.046651    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.046906 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.046686    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.046906 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.046755    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dfnb\" (UniqueName: \"kubernetes.io/projected/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kube-api-access-5dfnb\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.046906 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.046805    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148084 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148044    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148317 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148094    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148317 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148117    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148317 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148144    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148317 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148161    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148317 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148281    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148599 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148321    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148599 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148386    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dfnb\" (UniqueName: \"kubernetes.io/projected/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kube-api-access-5dfnb\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148599 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148555    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148722 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148604    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148722 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148689    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.148867 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.148848    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.149169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.149149    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.150676 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.150655    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.150867 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.150848    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.158159 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.158133    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dfnb\" (UniqueName: \"kubernetes.io/projected/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kube-api-access-5dfnb\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.245442 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.245327    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:44:54.384895 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.384871    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"]
Jul 07 02:44:54.386823 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:44:54.386797    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27eafe0c_a13c_4d62_9b9a_2fd263bc4adc.slice/crio-02ffac7d6ac77b5b3acca1423327256db89607b80114c0c779bcc36bfda252b7 WatchSource:0}: Error finding container 02ffac7d6ac77b5b3acca1423327256db89607b80114c0c779bcc36bfda252b7: Status 404 returned error can't find the container with id 02ffac7d6ac77b5b3acca1423327256db89607b80114c0c779bcc36bfda252b7
Jul 07 02:44:54.894926 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.894890    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" event={"ID":"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc","Type":"ContainerStarted","Data":"484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc"}
Jul 07 02:44:54.895133 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:54.894949    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" event={"ID":"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc","Type":"ContainerStarted","Data":"02ffac7d6ac77b5b3acca1423327256db89607b80114c0c779bcc36bfda252b7"}
Jul 07 02:44:58.916757 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:58.916719    2543 generic.go:356] "Generic (PLEG): container finished" podID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerID="484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc" exitCode=0
Jul 07 02:44:58.917142 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:58.916802    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" event={"ID":"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc","Type":"ContainerDied","Data":"484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc"}
Jul 07 02:44:59.924256 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:59.924221    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" event={"ID":"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc","Type":"ContainerStarted","Data":"9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4"}
Jul 07 02:44:59.952450 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:44:59.952387    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" podStartSLOduration=3.308700408 podStartE2EDuration="6.95236497s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.643664562s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:44:59.950802578 +0000 UTC m=+1612.593963100" watchObservedRunningTime="2026-07-07 02:44:59.95236497 +0000 UTC m=+1612.595525499"
Jul 07 02:45:00.246305 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:45:00.246208    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:45:04.246442 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:45:04.246400    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:45:04.247940 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:45:04.247913    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 07 02:45:14.246592 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:45:14.246534    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 07 02:45:24.246650 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:45:24.246601    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 07 02:45:34.246354 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:45:34.246297    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 07 02:45:44.246014 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:45:44.245958    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 07 02:45:54.246455 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:45:54.246409    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 07 02:46:04.246857 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:04.246786    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 07 02:46:14.255981 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:14.255942    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:46:14.263842 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:14.263817    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:46:15.967066 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:15.967024    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"]
Jul 07 02:46:30.992501 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:30.992409    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main" containerID="cri-o://9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4" gracePeriod=45
Jul 07 02:46:31.257305 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.257174    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 07 02:46:31.545873 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.545851    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:46:31.632559 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.632523    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-home\") pod \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") "
Jul 07 02:46:31.632559 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.632567    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kserve-provision-location\") pod \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") "
Jul 07 02:46:31.632825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.632606    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tmp-dir\") pod \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") "
Jul 07 02:46:31.632825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.632647    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-model-cache\") pod \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") "
Jul 07 02:46:31.632825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.632684    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-cabundle-cert\") pod \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") "
Jul 07 02:46:31.632825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.632737    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tls-certs\") pod \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") "
Jul 07 02:46:31.632825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.632759    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-dshm\") pod \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") "
Jul 07 02:46:31.632825 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.632819    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dfnb\" (UniqueName: \"kubernetes.io/projected/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kube-api-access-5dfnb\") pod \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\" (UID: \"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc\") "
Jul 07 02:46:31.633129 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.632941    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-model-cache" (OuterVolumeSpecName: "model-cache") pod "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" (UID: "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:46:31.633129 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.633048    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" (UID: "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:46:31.633129 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.633059    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-home" (OuterVolumeSpecName: "home") pod "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" (UID: "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:46:31.633129 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.633105    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:46:31.633479 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.633447    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" (UID: "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 02:46:31.635365 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.635287    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" (UID: "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:46:31.635707 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.635689    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-dshm" (OuterVolumeSpecName: "dshm") pod "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" (UID: "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:46:31.636437 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.636414    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kube-api-access-5dfnb" (OuterVolumeSpecName: "kube-api-access-5dfnb") pod "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" (UID: "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc"). InnerVolumeSpecName "kube-api-access-5dfnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:46:31.697865 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.697819    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" (UID: "27eafe0c-a13c-4d62-9b9a-2fd263bc4adc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:46:31.736780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.734670    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:46:31.736780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.734707    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-cabundle-cert\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:46:31.736780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.734727    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:46:31.736780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.734749    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:46:31.736780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.734763    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dfnb\" (UniqueName: \"kubernetes.io/projected/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kube-api-access-5dfnb\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:46:31.736780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.734776    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:46:31.736780 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:31.734822    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:46:32.367620 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.367581    2543 generic.go:356] "Generic (PLEG): container finished" podID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerID="9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4" exitCode=0
Jul 07 02:46:32.368045 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.367622    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" event={"ID":"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc","Type":"ContainerDied","Data":"9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4"}
Jul 07 02:46:32.368045 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.367660    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"
Jul 07 02:46:32.368045 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.367684    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84" event={"ID":"27eafe0c-a13c-4d62-9b9a-2fd263bc4adc","Type":"ContainerDied","Data":"02ffac7d6ac77b5b3acca1423327256db89607b80114c0c779bcc36bfda252b7"}
Jul 07 02:46:32.368045 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.367706    2543 scope.go:117] "RemoveContainer" containerID="9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4"
Jul 07 02:46:32.376643 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.376621    2543 scope.go:117] "RemoveContainer" containerID="484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc"
Jul 07 02:46:32.384944 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.384836    2543 scope.go:117] "RemoveContainer" containerID="9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4"
Jul 07 02:46:32.385212 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:46:32.385169    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4\": container with ID starting with 9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4 not found: ID does not exist" containerID="9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4"
Jul 07 02:46:32.385315 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.385225    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4"} err="failed to get container status \"9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4\": rpc error: code = NotFound desc = could not find container \"9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4\": container with ID starting with 9af27f02f4271b4ba077d61156be8e575f27f23043e3b2d88e3a1a27575db3c4 not found: ID does not exist"
Jul 07 02:46:32.385315 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.385250    2543 scope.go:117] "RemoveContainer" containerID="484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc"
Jul 07 02:46:32.385571 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:46:32.385547    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc\": container with ID starting with 484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc not found: ID does not exist" containerID="484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc"
Jul 07 02:46:32.385635 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.385583    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc"} err="failed to get container status \"484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc\": rpc error: code = NotFound desc = could not find container \"484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc\": container with ID starting with 484eddc863798e7a1bed2e953c6b78e970268a4c18014b2c014ee4ba07de21dc not found: ID does not exist"
Jul 07 02:46:32.387139 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.387117    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"]
Jul 07 02:46:32.390711 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:32.390691    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-64d79dc96c-48v84"]
Jul 07 02:46:33.979622 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:46:33.979585    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" path="/var/lib/kubelet/pods/27eafe0c-a13c-4d62-9b9a-2fd263bc4adc/volumes"
Jul 07 02:47:02.043963 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.043916    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"]
Jul 07 02:47:02.044611 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.044597    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main"
Jul 07 02:47:02.044676 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.044619    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main"
Jul 07 02:47:02.044676 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.044635    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="storage-initializer"
Jul 07 02:47:02.044676 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.044647    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="storage-initializer"
Jul 07 02:47:02.044817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.044760    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="27eafe0c-a13c-4d62-9b9a-2fd263bc4adc" containerName="main"
Jul 07 02:47:02.046611 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.046594    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.049388 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.049364    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:47:02.049503 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.049364    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 07 02:47:02.058591 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.058567    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"]
Jul 07 02:47:02.105181 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.105145    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgxh6\" (UniqueName: \"kubernetes.io/projected/1fcf5b34-cfd7-4be5-bd75-5491804ca826-kube-api-access-xgxh6\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.105181 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.105208    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.105418 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.105292    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.105418 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.105356    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.105418 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.105380    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.105418 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.105406    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.205801 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.205767    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.205990 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.205831    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.205990 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.205855    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.205990 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.205881    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.206153 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.206031    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgxh6\" (UniqueName: \"kubernetes.io/projected/1fcf5b34-cfd7-4be5-bd75-5491804ca826-kube-api-access-xgxh6\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.206153 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.206087    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.206317 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.206294    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.206382 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.206309    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.206495 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.206476    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.208255 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.208237    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.208525 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.208504    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:02.214637 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:02.214614    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgxh6\" (UniqueName: \"kubernetes.io/projected/1fcf5b34-cfd7-4be5-bd75-5491804ca826-kube-api-access-xgxh6\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:04.824810 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:04.824776    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:04.830763 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:04.830737    2543 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:04.925919 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:04.925877    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:04.926117 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:04.925999    2543 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:04.928742 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:04.928725    2543 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") DevicePath \"csi-07b3fce45dd55a8374628a957b64abbcd929ffdd3ca0d842186fd5ff05b726f2\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:05.025299 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:05.025266    2543 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4d4e79e722613b74efd0a3ee1474df31ab6a7717dd4a75fe1631c9eba9ff0a8b/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:05.043862 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:05.043831    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:05.058610 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:05.058572    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:05.199587 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:05.199558    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"]
Jul 07 02:47:05.514822 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:05.514786    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" event={"ID":"1fcf5b34-cfd7-4be5-bd75-5491804ca826","Type":"ContainerStarted","Data":"bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7"}
Jul 07 02:47:05.515004 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:05.514838    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" event={"ID":"1fcf5b34-cfd7-4be5-bd75-5491804ca826","Type":"ContainerStarted","Data":"81717e9253c7651c85b67f59cb278b398fd38dd59047753d0a7b64744824673e"}
Jul 07 02:47:05.534938 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:05.534883    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podStartSLOduration=3.534866164 podStartE2EDuration="3.534866164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:47:05.533475391 +0000 UTC m=+1738.176635917" watchObservedRunningTime="2026-07-07 02:47:05.534866164 +0000 UTC m=+1738.178026685"
Jul 07 02:47:06.059622 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:06.059586    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:15.059481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:15.059443    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:47:15.061156 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:15.061121    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:47:25.058999 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:25.058959    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:47:35.059076 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:35.059039    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:47:45.059583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:45.059536    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:47:55.059579 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:47:55.059538    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:48:05.059751 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:48:05.059702    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:48:07.947085 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:48:07.947058    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:48:07.952784 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:48:07.952762    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:48:15.059542 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:48:15.059499    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:48:25.059931 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:48:25.059886    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:48:35.059347 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:48:35.059292    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:48:45.059155 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:48:45.059115    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:48:55.059520 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:48:55.059472    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:49:05.069924 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:05.069890    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:49:05.077781 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:05.077757    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:49:09.595520 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:09.595487    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"]
Jul 07 02:49:24.619250 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:24.619162    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" containerID="cri-o://bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7" gracePeriod=45
Jul 07 02:49:25.071586 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.071473    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": dial tcp 10.134.0.43:8000: connect: connection refused"
Jul 07 02:49:25.375284 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.375255    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:49:25.525411 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.525374    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgxh6\" (UniqueName: \"kubernetes.io/projected/1fcf5b34-cfd7-4be5-bd75-5491804ca826-kube-api-access-xgxh6\") pod \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") "
Jul 07 02:49:25.525613 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.525455    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-model-cache\") pod \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") "
Jul 07 02:49:25.525613 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.525489    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-dshm\") pod \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") "
Jul 07 02:49:25.525613 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.525513    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tls-certs\") pod \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") "
Jul 07 02:49:25.525613 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.525531    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-home\") pod \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") "
Jul 07 02:49:25.525613 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.525582    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tmp-dir\") pod \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") "
Jul 07 02:49:25.525888 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.525698    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\" (UID: \"1fcf5b34-cfd7-4be5-bd75-5491804ca826\") "
Jul 07 02:49:25.525888 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.525729    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-model-cache" (OuterVolumeSpecName: "model-cache") pod "1fcf5b34-cfd7-4be5-bd75-5491804ca826" (UID: "1fcf5b34-cfd7-4be5-bd75-5491804ca826"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:49:25.525986 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.525957    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:49:25.526234 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.526209    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-home" (OuterVolumeSpecName: "home") pod "1fcf5b34-cfd7-4be5-bd75-5491804ca826" (UID: "1fcf5b34-cfd7-4be5-bd75-5491804ca826"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:49:25.527801 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.527774    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcf5b34-cfd7-4be5-bd75-5491804ca826-kube-api-access-xgxh6" (OuterVolumeSpecName: "kube-api-access-xgxh6") pod "1fcf5b34-cfd7-4be5-bd75-5491804ca826" (UID: "1fcf5b34-cfd7-4be5-bd75-5491804ca826"). InnerVolumeSpecName "kube-api-access-xgxh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:49:25.528293 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.528269    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1fcf5b34-cfd7-4be5-bd75-5491804ca826" (UID: "1fcf5b34-cfd7-4be5-bd75-5491804ca826"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:49:25.528293 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.528276    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-dshm" (OuterVolumeSpecName: "dshm") pod "1fcf5b34-cfd7-4be5-bd75-5491804ca826" (UID: "1fcf5b34-cfd7-4be5-bd75-5491804ca826"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:49:25.530610 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.530591    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8" (OuterVolumeSpecName: "kserve-pvc-source") pod "1fcf5b34-cfd7-4be5-bd75-5491804ca826" (UID: "1fcf5b34-cfd7-4be5-bd75-5491804ca826"). InnerVolumeSpecName "pvc-72dabf56-560c-4abb-87d3-214aad97d3e0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 07 02:49:25.537454 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.537430    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1fcf5b34-cfd7-4be5-bd75-5491804ca826" (UID: "1fcf5b34-cfd7-4be5-bd75-5491804ca826"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:49:25.626635 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.626606    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:49:25.626635 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.626635    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:49:25.627027 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.626646    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:49:25.627027 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.626653    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fcf5b34-cfd7-4be5-bd75-5491804ca826-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:49:25.627027 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.626685    2543 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") on node \"ip-10-0-132-101.ec2.internal\" "
Jul 07 02:49:25.627027 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.626695    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgxh6\" (UniqueName: \"kubernetes.io/projected/1fcf5b34-cfd7-4be5-bd75-5491804ca826-kube-api-access-xgxh6\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:49:25.655156 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.655137    2543 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-72dabf56-560c-4abb-87d3-214aad97d3e0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8") on node "ip-10-0-132-101.ec2.internal"
Jul 07 02:49:25.727737 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:25.727707    2543 reconciler_common.go:299] "Volume detached for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"csi-07b3fce45dd55a8374628a957b64abbcd929ffdd3ca0d842186fd5ff05b726f2\""
Jul 07 02:49:26.199337 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:26.199305    2543 generic.go:356] "Generic (PLEG): container finished" podID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerID="bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7" exitCode=0
Jul 07 02:49:26.199520 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:26.199354    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" event={"ID":"1fcf5b34-cfd7-4be5-bd75-5491804ca826","Type":"ContainerDied","Data":"bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7"}
Jul 07 02:49:26.199520 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:26.199363    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"
Jul 07 02:49:26.199520 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:26.199402    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9" event={"ID":"1fcf5b34-cfd7-4be5-bd75-5491804ca826","Type":"ContainerDied","Data":"81717e9253c7651c85b67f59cb278b398fd38dd59047753d0a7b64744824673e"}
Jul 07 02:49:26.199520 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:26.199425    2543 scope.go:117] "RemoveContainer" containerID="bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7"
Jul 07 02:49:26.208299 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:26.208277    2543 scope.go:117] "RemoveContainer" containerID="bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7"
Jul 07 02:49:26.208544 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:49:26.208524    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7\": container with ID starting with bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7 not found: ID does not exist" containerID="bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7"
Jul 07 02:49:26.208603 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:26.208551    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7"} err="failed to get container status \"bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7\": rpc error: code = NotFound desc = could not find container \"bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7\": container with ID starting with bb0fddf3c517ed542624e76d7f546db71b31c9b56611d0a1eab908edfdf1d3d7 not found: ID does not exist"
Jul 07 02:49:26.218118 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:26.218093    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"]
Jul 07 02:49:26.222313 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:26.222292    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dk7qh9"]
Jul 07 02:49:27.979837 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:27.979762    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" path="/var/lib/kubelet/pods/1fcf5b34-cfd7-4be5-bd75-5491804ca826/volumes"
Jul 07 02:49:33.488927 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.488884    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"]
Jul 07 02:49:33.490320 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.489271    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main"
Jul 07 02:49:33.490320 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.489285    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main"
Jul 07 02:49:33.490320 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.489350    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcf5b34-cfd7-4be5-bd75-5491804ca826" containerName="main"
Jul 07 02:49:33.491579 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.491558    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.493070 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.493047    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"]
Jul 07 02:49:33.494895 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.494876    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-6n8ms\""
Jul 07 02:49:33.495129 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.495113    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 07 02:49:33.495205 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.495122    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.495205 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.495172    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:49:33.508295 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.508274    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"]
Jul 07 02:49:33.510927 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.510903    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"]
Jul 07 02:49:33.592420 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.592389    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.592623 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.592455    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.592623 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.592521    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.592623 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.592556    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.592772 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.592639    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.693586 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693555    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.693797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693605    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2f9j\" (UniqueName: \"kubernetes.io/projected/5e892482-8f39-46a2-9e27-0ce2dfddafbe-kube-api-access-p2f9j\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.693797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693642    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.693797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693672    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.693797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693695    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a281c519-cecf-409d-a665-05c3b5c91081-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.693797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693730    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.693797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693767    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.693797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693787    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.694169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693856    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.694169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693889    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.694169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693923    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxn2\" (UniqueName: \"kubernetes.io/projected/a281c519-cecf-409d-a665-05c3b5c91081-kube-api-access-6hxn2\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.694169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.693977    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.694523 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.694242    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.694523 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.694241    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.694523 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.694294    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.696107 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.696087    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.696496 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.696475    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.794507 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.794433    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2f9j\" (UniqueName: \"kubernetes.io/projected/5e892482-8f39-46a2-9e27-0ce2dfddafbe-kube-api-access-p2f9j\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.794507 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.794481    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a281c519-cecf-409d-a665-05c3b5c91081-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.794724 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.794509    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.794724 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.794531    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.794724 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.794570    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxn2\" (UniqueName: \"kubernetes.io/projected/a281c519-cecf-409d-a665-05c3b5c91081-kube-api-access-6hxn2\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.794724 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.794607    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.794936 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.794794    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.794936 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.794923    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.795027 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.795001    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.795113 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.795096    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.797010 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.796988    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.797130 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.797086    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a281c519-cecf-409d-a665-05c3b5c91081-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:33.804682 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.804656    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2f9j\" (UniqueName: \"kubernetes.io/projected/5e892482-8f39-46a2-9e27-0ce2dfddafbe-kube-api-access-p2f9j\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:33.805072 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:33.805051    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxn2\" (UniqueName: \"kubernetes.io/projected/a281c519-cecf-409d-a665-05c3b5c91081-kube-api-access-6hxn2\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:38.643087 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.643045    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:38.648903 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.648868    2543 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:38.744573 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.744528    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:38.744758 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.744680    2543 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:38.748121 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.748100    2543 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") DevicePath \"csi-07b3fce45dd55a8374628a957b64abbcd929ffdd3ca0d842186fd5ff05b726f2\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:38.837668 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.837633    2543 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4d4e79e722613b74efd0a3ee1474df31ab6a7717dd4a75fe1631c9eba9ff0a8b/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:38.854321 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.854294    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:38.908876 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.908803    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:38.946716 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.946686    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:38.946867 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.946836    2543 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") DevicePath \"csi-07b3fce45dd55a8374628a957b64abbcd929ffdd3ca0d842186fd5ff05b726f2\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:38.951362 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.951341    2543 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") DevicePath \"csi-07b3fce45dd55a8374628a957b64abbcd929ffdd3ca0d842186fd5ff05b726f2\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:38.966295 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:38.966265    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:39.054521 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:39.054495    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"]
Jul 07 02:49:39.056650 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:49:39.056620    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e892482_8f39_46a2_9e27_0ce2dfddafbe.slice/crio-2b23760edaac4b2ac3fbf8918be0b34dd9802c615f539c96395aa638d503df22 WatchSource:0}: Error finding container 2b23760edaac4b2ac3fbf8918be0b34dd9802c615f539c96395aa638d503df22: Status 404 returned error can't find the container with id 2b23760edaac4b2ac3fbf8918be0b34dd9802c615f539c96395aa638d503df22
Jul 07 02:49:39.202096 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:39.201997    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:39.263034 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:39.262999    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" event={"ID":"5e892482-8f39-46a2-9e27-0ce2dfddafbe","Type":"ContainerStarted","Data":"46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5"}
Jul 07 02:49:39.263216 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:39.263060    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" event={"ID":"5e892482-8f39-46a2-9e27-0ce2dfddafbe","Type":"ContainerStarted","Data":"2b23760edaac4b2ac3fbf8918be0b34dd9802c615f539c96395aa638d503df22"}
Jul 07 02:49:39.283236 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:39.283158    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podStartSLOduration=6.283143013 podStartE2EDuration="6.283143013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:49:39.281407658 +0000 UTC m=+1891.924568215" watchObservedRunningTime="2026-07-07 02:49:39.283143013 +0000 UTC m=+1891.926303536"
Jul 07 02:49:39.347419 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:39.347392    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"]
Jul 07 02:49:39.350026 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:49:39.349993    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda281c519_cecf_409d_a665_05c3b5c91081.slice/crio-04f38181038c6f7bee0f86e5b17b172e2deba9adde2760fb487bbce2f9d566c1 WatchSource:0}: Error finding container 04f38181038c6f7bee0f86e5b17b172e2deba9adde2760fb487bbce2f9d566c1: Status 404 returned error can't find the container with id 04f38181038c6f7bee0f86e5b17b172e2deba9adde2760fb487bbce2f9d566c1
Jul 07 02:49:39.352127 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:39.352110    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 02:49:40.270228 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:40.270116    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" event={"ID":"a281c519-cecf-409d-a665-05c3b5c91081","Type":"ContainerStarted","Data":"d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f"}
Jul 07 02:49:40.270678 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:40.270447    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" event={"ID":"a281c519-cecf-409d-a665-05c3b5c91081","Type":"ContainerStarted","Data":"04f38181038c6f7bee0f86e5b17b172e2deba9adde2760fb487bbce2f9d566c1"}
Jul 07 02:49:40.271480 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:40.271375    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:41.278328 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:41.278298    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" event={"ID":"a281c519-cecf-409d-a665-05c3b5c91081","Type":"ContainerStarted","Data":"1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5"}
Jul 07 02:49:41.303402 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:41.303333    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podStartSLOduration=7.562035375 podStartE2EDuration="8.303315084s" totalImagesPullingTime="741.279709ms" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:49:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:49:41.302059375 +0000 UTC m=+1893.945219897" watchObservedRunningTime="2026-07-07 02:49:41.303315084 +0000 UTC m=+1893.946475643"
Jul 07 02:49:48.909235 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:48.909180    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:48.909872 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:48.909283    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:49:48.910719 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:48.910686    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:49:49.202253 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:49.202127    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:49.202253 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:49.202219    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:49:49.203837 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:49.203804    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:49:58.909717 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:58.909636    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:49:59.203087 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:59.202976    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:49:59.215692 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:49:59.215665    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:50:08.909873 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:08.909811    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:50:09.202629 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:09.202524    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:50:18.909408 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:18.909365    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:50:19.203124 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:19.202982    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:50:28.909400 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:28.909337    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:50:29.203124 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:29.203015    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:50:38.910141 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:38.910100    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:50:39.202803 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:39.202681    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:50:48.909168 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:48.909113    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:50:49.203218 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:49.203087    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:50:58.909912 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:58.909821    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:50:59.203268 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:50:59.203138    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:51:08.910060 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:08.910004    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:51:09.202968 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:09.202846    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:51:18.909421 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:18.909339    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:51:19.203354 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:19.203239    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:51:28.909384 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:28.909343    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:51:29.202863 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:29.202725    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:51:38.910163 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:38.910118    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:51:39.203481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:39.203370    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:51:48.909477 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:48.909424    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:51:49.202862 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:49.202751    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:51:58.909835 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:58.909792    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:51:59.202617 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:51:59.202506    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:52:08.910105 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:08.910055    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": dial tcp 10.134.0.45:8000: connect: connection refused"
Jul 07 02:52:09.203506 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:09.203388    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8001/health\": dial tcp 10.134.0.44:8001: connect: connection refused"
Jul 07 02:52:18.919117 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:18.919083    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:52:18.926525 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:18.926505    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:52:19.212509 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:19.212433    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:52:19.225021 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:19.224997    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:52:27.652818 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:27.652784    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"]
Jul 07 02:52:27.667477 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:27.667446    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"]
Jul 07 02:52:42.677638 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:42.677569    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main" containerID="cri-o://46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5" gracePeriod=45
Jul 07 02:52:42.692445 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:42.692384    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main" containerID="cri-o://1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5" gracePeriod=45
Jul 07 02:52:43.509817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.509782    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="llm-d-routing-sidecar" containerID="cri-o://d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f" gracePeriod=45
Jul 07 02:52:43.635663 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.635637    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:52:43.753737 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.753708    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:52:43.767054 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.767034    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2f9j\" (UniqueName: \"kubernetes.io/projected/5e892482-8f39-46a2-9e27-0ce2dfddafbe-kube-api-access-p2f9j\") pod \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") "
Jul 07 02:52:43.767136 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.767126    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-home\") pod \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") "
Jul 07 02:52:43.767180 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.767142    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-dshm\") pod \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") "
Jul 07 02:52:43.767180 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.767177    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tmp-dir\") pod \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") "
Jul 07 02:52:43.767303 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.767289    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") "
Jul 07 02:52:43.767355 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.767322    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-model-cache\") pod \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") "
Jul 07 02:52:43.767355 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.767340    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tls-certs\") pod \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\" (UID: \"5e892482-8f39-46a2-9e27-0ce2dfddafbe\") "
Jul 07 02:52:43.767657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.767630    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e892482-8f39-46a2-9e27-0ce2dfddafbe" (UID: "5e892482-8f39-46a2-9e27-0ce2dfddafbe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:52:43.768004 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.767978    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-home" (OuterVolumeSpecName: "home") pod "5e892482-8f39-46a2-9e27-0ce2dfddafbe" (UID: "5e892482-8f39-46a2-9e27-0ce2dfddafbe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:52:43.769554 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.769526    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e892482-8f39-46a2-9e27-0ce2dfddafbe-kube-api-access-p2f9j" (OuterVolumeSpecName: "kube-api-access-p2f9j") pod "5e892482-8f39-46a2-9e27-0ce2dfddafbe" (UID: "5e892482-8f39-46a2-9e27-0ce2dfddafbe"). InnerVolumeSpecName "kube-api-access-p2f9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:52:43.769694 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.769666    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-dshm" (OuterVolumeSpecName: "dshm") pod "5e892482-8f39-46a2-9e27-0ce2dfddafbe" (UID: "5e892482-8f39-46a2-9e27-0ce2dfddafbe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:52:43.769775 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.769685    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e892482-8f39-46a2-9e27-0ce2dfddafbe" (UID: "5e892482-8f39-46a2-9e27-0ce2dfddafbe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:52:43.770593 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.770573    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8" (OuterVolumeSpecName: "kserve-pvc-source") pod "5e892482-8f39-46a2-9e27-0ce2dfddafbe" (UID: "5e892482-8f39-46a2-9e27-0ce2dfddafbe"). InnerVolumeSpecName "pvc-72dabf56-560c-4abb-87d3-214aad97d3e0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 07 02:52:43.780066 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.780045    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e892482-8f39-46a2-9e27-0ce2dfddafbe" (UID: "5e892482-8f39-46a2-9e27-0ce2dfddafbe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:52:43.867943 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.867909    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-dshm\") pod \"a281c519-cecf-409d-a665-05c3b5c91081\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") "
Jul 07 02:52:43.868115 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.867972    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hxn2\" (UniqueName: \"kubernetes.io/projected/a281c519-cecf-409d-a665-05c3b5c91081-kube-api-access-6hxn2\") pod \"a281c519-cecf-409d-a665-05c3b5c91081\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") "
Jul 07 02:52:43.868115 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868059    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"a281c519-cecf-409d-a665-05c3b5c91081\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") "
Jul 07 02:52:43.868115 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868083    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-tmp-dir\") pod \"a281c519-cecf-409d-a665-05c3b5c91081\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") "
Jul 07 02:52:43.868284 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868114    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-model-cache\") pod \"a281c519-cecf-409d-a665-05c3b5c91081\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") "
Jul 07 02:52:43.868284 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868155    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a281c519-cecf-409d-a665-05c3b5c91081-tls-certs\") pod \"a281c519-cecf-409d-a665-05c3b5c91081\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") "
Jul 07 02:52:43.868284 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868216    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-home\") pod \"a281c519-cecf-409d-a665-05c3b5c91081\" (UID: \"a281c519-cecf-409d-a665-05c3b5c91081\") "
Jul 07 02:52:43.868485 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868439    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-model-cache" (OuterVolumeSpecName: "model-cache") pod "a281c519-cecf-409d-a665-05c3b5c91081" (UID: "a281c519-cecf-409d-a665-05c3b5c91081"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:52:43.868583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868491    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.868583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868508    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.868583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868519    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.868583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868532    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e892482-8f39-46a2-9e27-0ce2dfddafbe-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.868583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868544    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e892482-8f39-46a2-9e27-0ce2dfddafbe-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.868583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.868557    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2f9j\" (UniqueName: \"kubernetes.io/projected/5e892482-8f39-46a2-9e27-0ce2dfddafbe-kube-api-access-p2f9j\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.869253 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.869229    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-home" (OuterVolumeSpecName: "home") pod "a281c519-cecf-409d-a665-05c3b5c91081" (UID: "a281c519-cecf-409d-a665-05c3b5c91081"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:52:43.870522 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.870496    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a281c519-cecf-409d-a665-05c3b5c91081-kube-api-access-6hxn2" (OuterVolumeSpecName: "kube-api-access-6hxn2") pod "a281c519-cecf-409d-a665-05c3b5c91081" (UID: "a281c519-cecf-409d-a665-05c3b5c91081"). InnerVolumeSpecName "kube-api-access-6hxn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:52:43.870522 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.870526    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-dshm" (OuterVolumeSpecName: "dshm") pod "a281c519-cecf-409d-a665-05c3b5c91081" (UID: "a281c519-cecf-409d-a665-05c3b5c91081"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:52:43.870797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.870781    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a281c519-cecf-409d-a665-05c3b5c91081-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a281c519-cecf-409d-a665-05c3b5c91081" (UID: "a281c519-cecf-409d-a665-05c3b5c91081"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:52:43.871559 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.871542    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8" (OuterVolumeSpecName: "kserve-pvc-source") pod "a281c519-cecf-409d-a665-05c3b5c91081" (UID: "a281c519-cecf-409d-a665-05c3b5c91081"). InnerVolumeSpecName "pvc-72dabf56-560c-4abb-87d3-214aad97d3e0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 07 02:52:43.880082 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.880059    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a281c519-cecf-409d-a665-05c3b5c91081" (UID: "a281c519-cecf-409d-a665-05c3b5c91081"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:52:43.969656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.969565    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.969656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.969598    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a281c519-cecf-409d-a665-05c3b5c91081-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.969656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.969607    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.969656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.969614    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.969656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.969623    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxn2\" (UniqueName: \"kubernetes.io/projected/a281c519-cecf-409d-a665-05c3b5c91081-kube-api-access-6hxn2\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:43.969656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.969653    2543 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") on node \"ip-10-0-132-101.ec2.internal\" "
Jul 07 02:52:43.969656 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:43.969662    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a281c519-cecf-409d-a665-05c3b5c91081-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:52:44.001740 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.001723    2543 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-72dabf56-560c-4abb-87d3-214aad97d3e0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8") on node "ip-10-0-132-101.ec2.internal"
Jul 07 02:52:44.070337 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.070303    2543 reconciler_common.go:299] "Volume detached for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"csi-07b3fce45dd55a8374628a957b64abbcd929ffdd3ca0d842186fd5ff05b726f2\""
Jul 07 02:52:44.217964 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.217927    2543 generic.go:356] "Generic (PLEG): container finished" podID="a281c519-cecf-409d-a665-05c3b5c91081" containerID="1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5" exitCode=0
Jul 07 02:52:44.217964 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.217958    2543 generic.go:356] "Generic (PLEG): container finished" podID="a281c519-cecf-409d-a665-05c3b5c91081" containerID="d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f" exitCode=0
Jul 07 02:52:44.218198 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.217987    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"
Jul 07 02:52:44.218198 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.218009    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" event={"ID":"a281c519-cecf-409d-a665-05c3b5c91081","Type":"ContainerDied","Data":"1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5"}
Jul 07 02:52:44.218198 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.218063    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" event={"ID":"a281c519-cecf-409d-a665-05c3b5c91081","Type":"ContainerDied","Data":"d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f"}
Jul 07 02:52:44.218198 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.218082    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs" event={"ID":"a281c519-cecf-409d-a665-05c3b5c91081","Type":"ContainerDied","Data":"04f38181038c6f7bee0f86e5b17b172e2deba9adde2760fb487bbce2f9d566c1"}
Jul 07 02:52:44.218198 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.218111    2543 scope.go:117] "RemoveContainer" containerID="1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5"
Jul 07 02:52:44.219393 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.219375    2543 generic.go:356] "Generic (PLEG): container finished" podID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerID="46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5" exitCode=0
Jul 07 02:52:44.219464 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.219429    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"
Jul 07 02:52:44.219464 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.219449    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" event={"ID":"5e892482-8f39-46a2-9e27-0ce2dfddafbe","Type":"ContainerDied","Data":"46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5"}
Jul 07 02:52:44.219566 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.219490    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7" event={"ID":"5e892482-8f39-46a2-9e27-0ce2dfddafbe","Type":"ContainerDied","Data":"2b23760edaac4b2ac3fbf8918be0b34dd9802c615f539c96395aa638d503df22"}
Jul 07 02:52:44.227105 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.227086    2543 scope.go:117] "RemoveContainer" containerID="d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f"
Jul 07 02:52:44.235447 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.235430    2543 scope.go:117] "RemoveContainer" containerID="1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5"
Jul 07 02:52:44.235712 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:52:44.235692    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5\": container with ID starting with 1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5 not found: ID does not exist" containerID="1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5"
Jul 07 02:52:44.235774 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.235720    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5"} err="failed to get container status \"1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5\": rpc error: code = NotFound desc = could not find container \"1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5\": container with ID starting with 1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5 not found: ID does not exist"
Jul 07 02:52:44.235774 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.235742    2543 scope.go:117] "RemoveContainer" containerID="d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f"
Jul 07 02:52:44.236003 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:52:44.235985    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f\": container with ID starting with d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f not found: ID does not exist" containerID="d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f"
Jul 07 02:52:44.236077 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.236013    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f"} err="failed to get container status \"d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f\": rpc error: code = NotFound desc = could not find container \"d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f\": container with ID starting with d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f not found: ID does not exist"
Jul 07 02:52:44.236077 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.236033    2543 scope.go:117] "RemoveContainer" containerID="1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5"
Jul 07 02:52:44.236310 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.236292    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5"} err="failed to get container status \"1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5\": rpc error: code = NotFound desc = could not find container \"1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5\": container with ID starting with 1ad50531f219019287469e7ec8c22c7488851ea1061d7fb1d1dd67b96f2a1dc5 not found: ID does not exist"
Jul 07 02:52:44.236406 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.236312    2543 scope.go:117] "RemoveContainer" containerID="d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f"
Jul 07 02:52:44.236600 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.236573    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f"} err="failed to get container status \"d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f\": rpc error: code = NotFound desc = could not find container \"d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f\": container with ID starting with d3e0744ba22dfab413edb9a964d5267f4edbdf56261a97fa548e27a825df629f not found: ID does not exist"
Jul 07 02:52:44.236643 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.236604    2543 scope.go:117] "RemoveContainer" containerID="46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5"
Jul 07 02:52:44.245547 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.245526    2543 scope.go:117] "RemoveContainer" containerID="46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5"
Jul 07 02:52:44.245865 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:52:44.245840    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5\": container with ID starting with 46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5 not found: ID does not exist" containerID="46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5"
Jul 07 02:52:44.245975 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.245872    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5"} err="failed to get container status \"46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5\": rpc error: code = NotFound desc = could not find container \"46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5\": container with ID starting with 46786b2fcf43d7e1fd4132df769800586967c0390958c5821a8177c0d5af96c5 not found: ID does not exist"
Jul 07 02:52:44.246614 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.246592    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"]
Jul 07 02:52:44.248152 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.248134    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79lbjf7"]
Jul 07 02:52:44.282618 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.282591    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"]
Jul 07 02:52:44.296684 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:44.296653    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-72gxs"]
Jul 07 02:52:45.979874 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:45.979839    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" path="/var/lib/kubelet/pods/5e892482-8f39-46a2-9e27-0ce2dfddafbe/volumes"
Jul 07 02:52:45.980283 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:45.980265    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a281c519-cecf-409d-a665-05c3b5c91081" path="/var/lib/kubelet/pods/a281c519-cecf-409d-a665-05c3b5c91081/volumes"
Jul 07 02:52:55.551645 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.551609    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 07 02:52:55.552038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.551958    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="llm-d-routing-sidecar"
Jul 07 02:52:55.552038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.551976    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="llm-d-routing-sidecar"
Jul 07 02:52:55.552038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.551987    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main"
Jul 07 02:52:55.552038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.551995    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main"
Jul 07 02:52:55.552038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.552019    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main"
Jul 07 02:52:55.552038 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.552024    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main"
Jul 07 02:52:55.552253 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.552093    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="llm-d-routing-sidecar"
Jul 07 02:52:55.552253 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.552103    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="a281c519-cecf-409d-a665-05c3b5c91081" containerName="main"
Jul 07 02:52:55.552253 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.552113    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e892482-8f39-46a2-9e27-0ce2dfddafbe" containerName="main"
Jul 07 02:52:55.556731 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.556710    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.563306 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.563280    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 02:52:55.563449 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.563400    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-vfp8j\""
Jul 07 02:52:55.564285 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.564255    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 07 02:52:55.574522 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.574497    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.574672 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.574549    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.574672 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.574608    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682f1053-c9ea-4561-8827-9fb7d405bfce-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.574672 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.574635    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.574672 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.574655    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.590758 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.590730    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 07 02:52:55.675558 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.675528    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.675759 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.675591    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.675759 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.675621    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvzf4\" (UniqueName: \"kubernetes.io/projected/682f1053-c9ea-4561-8827-9fb7d405bfce-kube-api-access-cvzf4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.675759 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.675667    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682f1053-c9ea-4561-8827-9fb7d405bfce-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.675759 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.675698    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.675759 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.675721    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.676015 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.675998    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.676077 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.676039    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.676077 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.676053    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.677892 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.677871    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.678101 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.678083    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682f1053-c9ea-4561-8827-9fb7d405bfce-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.727069 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.727036    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 07 02:52:55.730570 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.730544    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.749065 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.749036    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 07 02:52:55.776509 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.776476    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvzf4\" (UniqueName: \"kubernetes.io/projected/682f1053-c9ea-4561-8827-9fb7d405bfce-kube-api-access-cvzf4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.820902 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.820831    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvzf4\" (UniqueName: \"kubernetes.io/projected/682f1053-c9ea-4561-8827-9fb7d405bfce-kube-api-access-cvzf4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:52:55.877883 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.877798    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.877883 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.877881    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.878139 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.877960    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.878139 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.878025    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.878139 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.878087    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9knkh\" (UniqueName: \"kubernetes.io/projected/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-kube-api-access-9knkh\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.878139 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.878129    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.978729 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.978680    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9knkh\" (UniqueName: \"kubernetes.io/projected/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-kube-api-access-9knkh\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.978729 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.978733    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.978986 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.978781    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.978986 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.978809    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.978986 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.978839    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.978986 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.978885    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.979258 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.979232    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.979315 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.979256    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.979388 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.979373    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.981061 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.981040    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.981364 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.981347    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:52:55.996220 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:52:55.996169    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9knkh\" (UniqueName: \"kubernetes.io/projected/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-kube-api-access-9knkh\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:53:03.347482 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.347441    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:53:03.355238 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.355180    2543 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:53:03.449024 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.448988    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:53:03.449217 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.449138    2543 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:53:03.452483 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.452456    2543 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") DevicePath \"csi-07b3fce45dd55a8374628a957b64abbcd929ffdd3ca0d842186fd5ff05b726f2\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:53:03.543377 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.543340    2543 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/4d4e79e722613b74efd0a3ee1474df31ab6a7717dd4a75fe1631c9eba9ff0a8b/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:53:03.555499 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.555471    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:53:03.650325 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.650295    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:53:03.650489 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.650400    2543 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") DevicePath \"csi-07b3fce45dd55a8374628a957b64abbcd929ffdd3ca0d842186fd5ff05b726f2\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:53:03.654341 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.654321    2543 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") DevicePath \"csi-07b3fce45dd55a8374628a957b64abbcd929ffdd3ca0d842186fd5ff05b726f2\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:53:03.667832 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.667802    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:53:03.841487 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.841451    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:53:03.967710 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.967679    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:53:03.984946 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:03.984921    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 07 02:53:03.987104 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:53:03.987081    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50fec74_db2b_4a4e_b2a1_f76ec69fd868.slice/crio-e6f0c16a047bc09d9bacda669bf3ad5379edc46e3e3c5160c6680ac77438f9e3 WatchSource:0}: Error finding container e6f0c16a047bc09d9bacda669bf3ad5379edc46e3e3c5160c6680ac77438f9e3: Status 404 returned error can't find the container with id e6f0c16a047bc09d9bacda669bf3ad5379edc46e3e3c5160c6680ac77438f9e3
Jul 07 02:53:04.110706 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:04.110677    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 07 02:53:04.113421 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:53:04.113393    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod682f1053_c9ea_4561_8827_9fb7d405bfce.slice/crio-4603971d30770964a775e28abf2b3e62f6ade0a30712bbae8164592f31eaaef4 WatchSource:0}: Error finding container 4603971d30770964a775e28abf2b3e62f6ade0a30712bbae8164592f31eaaef4: Status 404 returned error can't find the container with id 4603971d30770964a775e28abf2b3e62f6ade0a30712bbae8164592f31eaaef4
Jul 07 02:53:04.314721 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:04.314619    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"682f1053-c9ea-4561-8827-9fb7d405bfce","Type":"ContainerStarted","Data":"9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d"}
Jul 07 02:53:04.314721 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:04.314680    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"682f1053-c9ea-4561-8827-9fb7d405bfce","Type":"ContainerStarted","Data":"4603971d30770964a775e28abf2b3e62f6ade0a30712bbae8164592f31eaaef4"}
Jul 07 02:53:04.316533 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:04.316502    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"d50fec74-db2b-4a4e-b2a1-f76ec69fd868","Type":"ContainerStarted","Data":"c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c"}
Jul 07 02:53:04.316657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:04.316560    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"d50fec74-db2b-4a4e-b2a1-f76ec69fd868","Type":"ContainerStarted","Data":"e6f0c16a047bc09d9bacda669bf3ad5379edc46e3e3c5160c6680ac77438f9e3"}
Jul 07 02:53:04.335898 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:04.335834    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=9.335811788 podStartE2EDuration="9.335811788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:53:04.33392646 +0000 UTC m=+2096.977086982" watchObservedRunningTime="2026-07-07 02:53:04.335811788 +0000 UTC m=+2096.978972311"
Jul 07 02:53:04.352678 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:04.352618    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=9.352600057 podStartE2EDuration="9.352600057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 02:52:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:53:04.351074859 +0000 UTC m=+2096.994235415" watchObservedRunningTime="2026-07-07 02:53:04.352600057 +0000 UTC m=+2096.995760579"
Jul 07 02:53:04.968219 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:04.968170    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:53:07.983301 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:07.983270    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:53:07.986618 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:07.986593    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:53:13.968158 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:13.968124    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:53:13.969940 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:13.969913    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:53:23.968943 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:23.968891    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:53:33.968628 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:33.968585    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:53:43.968577 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:43.968527    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:53:53.968567 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:53:53.968524    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:54:03.968345 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:54:03.968239    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:54:13.968199 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:54:13.968139    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:54:23.968900 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:54:23.968850    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:54:33.968682 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:54:33.968635    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:54:43.968181 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:54:43.968126    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:54:53.969060 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:54:53.969012    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:55:03.969062 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:55:03.969011    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:55:13.968521 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:55:13.968470    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/health\": dial tcp 10.134.0.46:8000: connect: connection refused"
Jul 07 02:55:23.980486 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:55:23.980457    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:55:23.988589 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:55:23.988568    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:56:00.762444 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.762404    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"]
Jul 07 02:56:00.766375 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.766355    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.768762 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.768740    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 02:56:00.768891 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.768797    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 07 02:56:00.777356 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.777330    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"]
Jul 07 02:56:00.861699 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.861665    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.861699 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.861700    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.861917 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.861721    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.861917 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.861746    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/548a3b24-ae47-47eb-86df-7807ec615923-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.861917 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.861790    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbrpz\" (UniqueName: \"kubernetes.io/projected/548a3b24-ae47-47eb-86df-7807ec615923-kube-api-access-wbrpz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.861917 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.861838    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.861917 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.861871    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963004 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.962973    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963004 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.963008    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963290 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.963029    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963290 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.963055    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/548a3b24-ae47-47eb-86df-7807ec615923-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963290 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.963075    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbrpz\" (UniqueName: \"kubernetes.io/projected/548a3b24-ae47-47eb-86df-7807ec615923-kube-api-access-wbrpz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963290 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.963123    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963290 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.963165    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963570 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.963499    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963570 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.963536    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963686 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.963673    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.963730 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.963670    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.965472 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.965452    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.965766 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.965748    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/548a3b24-ae47-47eb-86df-7807ec615923-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:00.974368 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:00.974342    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbrpz\" (UniqueName: \"kubernetes.io/projected/548a3b24-ae47-47eb-86df-7807ec615923-kube-api-access-wbrpz\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:01.076980 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:01.076899    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:01.222112 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:01.222090    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"]
Jul 07 02:56:01.224618 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:56:01.224591    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548a3b24_ae47_47eb_86df_7807ec615923.slice/crio-1c58c6cf8db6ac0f05533ba5f78fa20144b02689f91229741a105d0837864037 WatchSource:0}: Error finding container 1c58c6cf8db6ac0f05533ba5f78fa20144b02689f91229741a105d0837864037: Status 404 returned error can't find the container with id 1c58c6cf8db6ac0f05533ba5f78fa20144b02689f91229741a105d0837864037
Jul 07 02:56:02.222299 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:02.222246    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" event={"ID":"548a3b24-ae47-47eb-86df-7807ec615923","Type":"ContainerStarted","Data":"5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830"}
Jul 07 02:56:02.222785 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:02.222339    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" event={"ID":"548a3b24-ae47-47eb-86df-7807ec615923","Type":"ContainerStarted","Data":"1c58c6cf8db6ac0f05533ba5f78fa20144b02689f91229741a105d0837864037"}
Jul 07 02:56:05.242202 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:05.242098    2543 generic.go:356] "Generic (PLEG): container finished" podID="548a3b24-ae47-47eb-86df-7807ec615923" containerID="5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830" exitCode=0
Jul 07 02:56:05.242730 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:05.242178    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" event={"ID":"548a3b24-ae47-47eb-86df-7807ec615923","Type":"ContainerDied","Data":"5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830"}
Jul 07 02:56:06.251028 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:06.250975    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" event={"ID":"548a3b24-ae47-47eb-86df-7807ec615923","Type":"ContainerStarted","Data":"4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46"}
Jul 07 02:56:06.275935 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:06.275879    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podStartSLOduration=2.657318041 podStartE2EDuration="6.275863151s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.61854511s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:56:06.273266958 +0000 UTC m=+2278.916427473" watchObservedRunningTime="2026-07-07 02:56:06.275863151 +0000 UTC m=+2278.919023672"
Jul 07 02:56:07.077615 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:07.077580    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:11.078097 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:11.078055    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:56:11.079803 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:11.079773    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:56:18.157398 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:18.157348    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 07 02:56:21.078510 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:21.078460    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:56:31.077762 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:31.077708    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:56:33.183324 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:33.183261    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main" containerID="cri-o://9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d" gracePeriod=45
Jul 07 02:56:33.941266 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:33.941234    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:56:34.091037 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.091009    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-model-cache\") pod \"682f1053-c9ea-4561-8827-9fb7d405bfce\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") "
Jul 07 02:56:34.091037 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.091049    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-home\") pod \"682f1053-c9ea-4561-8827-9fb7d405bfce\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") "
Jul 07 02:56:34.091326 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.091082    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvzf4\" (UniqueName: \"kubernetes.io/projected/682f1053-c9ea-4561-8827-9fb7d405bfce-kube-api-access-cvzf4\") pod \"682f1053-c9ea-4561-8827-9fb7d405bfce\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") "
Jul 07 02:56:34.091326 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.091161    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-tmp-dir\") pod \"682f1053-c9ea-4561-8827-9fb7d405bfce\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") "
Jul 07 02:56:34.091326 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.091209    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682f1053-c9ea-4561-8827-9fb7d405bfce-tls-certs\") pod \"682f1053-c9ea-4561-8827-9fb7d405bfce\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") "
Jul 07 02:56:34.091326 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.091300    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-dshm\") pod \"682f1053-c9ea-4561-8827-9fb7d405bfce\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") "
Jul 07 02:56:34.091524 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.091407    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"682f1053-c9ea-4561-8827-9fb7d405bfce\" (UID: \"682f1053-c9ea-4561-8827-9fb7d405bfce\") "
Jul 07 02:56:34.091666 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.091634    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-model-cache" (OuterVolumeSpecName: "model-cache") pod "682f1053-c9ea-4561-8827-9fb7d405bfce" (UID: "682f1053-c9ea-4561-8827-9fb7d405bfce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:56:34.092040 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.091720    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-home" (OuterVolumeSpecName: "home") pod "682f1053-c9ea-4561-8827-9fb7d405bfce" (UID: "682f1053-c9ea-4561-8827-9fb7d405bfce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:56:34.093906 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.093875    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682f1053-c9ea-4561-8827-9fb7d405bfce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "682f1053-c9ea-4561-8827-9fb7d405bfce" (UID: "682f1053-c9ea-4561-8827-9fb7d405bfce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:56:34.094152 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.094121    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-dshm" (OuterVolumeSpecName: "dshm") pod "682f1053-c9ea-4561-8827-9fb7d405bfce" (UID: "682f1053-c9ea-4561-8827-9fb7d405bfce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:56:34.094152 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.094130    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682f1053-c9ea-4561-8827-9fb7d405bfce-kube-api-access-cvzf4" (OuterVolumeSpecName: "kube-api-access-cvzf4") pod "682f1053-c9ea-4561-8827-9fb7d405bfce" (UID: "682f1053-c9ea-4561-8827-9fb7d405bfce"). InnerVolumeSpecName "kube-api-access-cvzf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:56:34.094896 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.094873    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8" (OuterVolumeSpecName: "kserve-pvc-source") pod "682f1053-c9ea-4561-8827-9fb7d405bfce" (UID: "682f1053-c9ea-4561-8827-9fb7d405bfce"). InnerVolumeSpecName "pvc-72dabf56-560c-4abb-87d3-214aad97d3e0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 07 02:56:34.104443 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.104411    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "682f1053-c9ea-4561-8827-9fb7d405bfce" (UID: "682f1053-c9ea-4561-8827-9fb7d405bfce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:56:34.192809 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.192769    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:34.192809 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.192803    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:34.192809 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.192814    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:34.193288 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.192823    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvzf4\" (UniqueName: \"kubernetes.io/projected/682f1053-c9ea-4561-8827-9fb7d405bfce-kube-api-access-cvzf4\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:34.193288 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.192833    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/682f1053-c9ea-4561-8827-9fb7d405bfce-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:34.193288 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.192841    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/682f1053-c9ea-4561-8827-9fb7d405bfce-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:34.397283 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.397249    2543 generic.go:356] "Generic (PLEG): container finished" podID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerID="9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d" exitCode=0
Jul 07 02:56:34.397434 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.397313    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"682f1053-c9ea-4561-8827-9fb7d405bfce","Type":"ContainerDied","Data":"9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d"}
Jul 07 02:56:34.397434 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.397318    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 07 02:56:34.397434 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.397357    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"682f1053-c9ea-4561-8827-9fb7d405bfce","Type":"ContainerDied","Data":"4603971d30770964a775e28abf2b3e62f6ade0a30712bbae8164592f31eaaef4"}
Jul 07 02:56:34.397434 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.397377    2543 scope.go:117] "RemoveContainer" containerID="9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d"
Jul 07 02:56:34.406171 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.406151    2543 scope.go:117] "RemoveContainer" containerID="9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d"
Jul 07 02:56:34.406502 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:56:34.406480    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d\": container with ID starting with 9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d not found: ID does not exist" containerID="9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d"
Jul 07 02:56:34.406565 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.406511    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d"} err="failed to get container status \"9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d\": rpc error: code = NotFound desc = could not find container \"9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d\": container with ID starting with 9dc06d8d57a45ca85feda073c791a5494f040cac0af727f283ccd2105f45883d not found: ID does not exist"
Jul 07 02:56:34.429481 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.429443    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 07 02:56:34.433562 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.433533    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 07 02:56:34.470078 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:34.470046    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 07 02:56:35.980797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:35.980761    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" path="/var/lib/kubelet/pods/682f1053-c9ea-4561-8827-9fb7d405bfce/volumes"
Jul 07 02:56:41.077932 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:41.077876    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:56:49.492895 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:49.492855    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="d50fec74-db2b-4a4e-b2a1-f76ec69fd868" containerName="main" containerID="cri-o://c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c" gracePeriod=45
Jul 07 02:56:50.170103 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.170064    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:56:50.238730 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.238699    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") pod \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") "
Jul 07 02:56:50.238920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.238767    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-dshm\") pod \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") "
Jul 07 02:56:50.238920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.238810    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-home\") pod \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") "
Jul 07 02:56:50.238920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.238836    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tls-certs\") pod \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") "
Jul 07 02:56:50.238920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.238878    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9knkh\" (UniqueName: \"kubernetes.io/projected/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-kube-api-access-9knkh\") pod \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") "
Jul 07 02:56:50.238920 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.238913    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-model-cache\") pod \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") "
Jul 07 02:56:50.239251 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.238936    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tmp-dir\") pod \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\" (UID: \"d50fec74-db2b-4a4e-b2a1-f76ec69fd868\") "
Jul 07 02:56:50.239251 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.239225    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-model-cache" (OuterVolumeSpecName: "model-cache") pod "d50fec74-db2b-4a4e-b2a1-f76ec69fd868" (UID: "d50fec74-db2b-4a4e-b2a1-f76ec69fd868"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:56:50.240113 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.240085    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-home" (OuterVolumeSpecName: "home") pod "d50fec74-db2b-4a4e-b2a1-f76ec69fd868" (UID: "d50fec74-db2b-4a4e-b2a1-f76ec69fd868"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:56:50.241451 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.241425    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-dshm" (OuterVolumeSpecName: "dshm") pod "d50fec74-db2b-4a4e-b2a1-f76ec69fd868" (UID: "d50fec74-db2b-4a4e-b2a1-f76ec69fd868"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:56:50.241551 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.241489    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d50fec74-db2b-4a4e-b2a1-f76ec69fd868" (UID: "d50fec74-db2b-4a4e-b2a1-f76ec69fd868"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:56:50.241706 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.241684    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-kube-api-access-9knkh" (OuterVolumeSpecName: "kube-api-access-9knkh") pod "d50fec74-db2b-4a4e-b2a1-f76ec69fd868" (UID: "d50fec74-db2b-4a4e-b2a1-f76ec69fd868"). InnerVolumeSpecName "kube-api-access-9knkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:56:50.242362 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.242330    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8" (OuterVolumeSpecName: "kserve-pvc-source") pod "d50fec74-db2b-4a4e-b2a1-f76ec69fd868" (UID: "d50fec74-db2b-4a4e-b2a1-f76ec69fd868"). InnerVolumeSpecName "pvc-72dabf56-560c-4abb-87d3-214aad97d3e0". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 07 02:56:50.250132 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.250108    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d50fec74-db2b-4a4e-b2a1-f76ec69fd868" (UID: "d50fec74-db2b-4a4e-b2a1-f76ec69fd868"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:56:50.339909 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.339855    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:50.339909 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.339901    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:50.339909 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.339913    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:50.339909 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.339925    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9knkh\" (UniqueName: \"kubernetes.io/projected/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-kube-api-access-9knkh\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:50.340232 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.339934    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:50.340232 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.339943    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d50fec74-db2b-4a4e-b2a1-f76ec69fd868-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:56:50.340232 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.339971    2543 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") on node \"ip-10-0-132-101.ec2.internal\" "
Jul 07 02:56:50.369252 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.369225    2543 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-72dabf56-560c-4abb-87d3-214aad97d3e0" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8") on node "ip-10-0-132-101.ec2.internal"
Jul 07 02:56:50.440984 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.440947    2543 reconciler_common.go:299] "Volume detached for volume \"pvc-72dabf56-560c-4abb-87d3-214aad97d3e0\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-03c13f35e503826e8\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"csi-07b3fce45dd55a8374628a957b64abbcd929ffdd3ca0d842186fd5ff05b726f2\""
Jul 07 02:56:50.475232 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.475160    2543 generic.go:356] "Generic (PLEG): container finished" podID="d50fec74-db2b-4a4e-b2a1-f76ec69fd868" containerID="c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c" exitCode=0
Jul 07 02:56:50.475232 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.475230    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 07 02:56:50.475426 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.475225    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"d50fec74-db2b-4a4e-b2a1-f76ec69fd868","Type":"ContainerDied","Data":"c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c"}
Jul 07 02:56:50.475426 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.475360    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"d50fec74-db2b-4a4e-b2a1-f76ec69fd868","Type":"ContainerDied","Data":"e6f0c16a047bc09d9bacda669bf3ad5379edc46e3e3c5160c6680ac77438f9e3"}
Jul 07 02:56:50.475426 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.475386    2543 scope.go:117] "RemoveContainer" containerID="c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c"
Jul 07 02:56:50.491424 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.491391    2543 scope.go:117] "RemoveContainer" containerID="c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c"
Jul 07 02:56:50.491823 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:56:50.491795    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c\": container with ID starting with c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c not found: ID does not exist" containerID="c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c"
Jul 07 02:56:50.491913 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.491833    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c"} err="failed to get container status \"c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c\": rpc error: code = NotFound desc = could not find container \"c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c\": container with ID starting with c4b266236ff5619a2ad479f6c440a3f45a9567ff16eabf4899102a0b2058cb4c not found: ID does not exist"
Jul 07 02:56:50.502413 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.502353    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 07 02:56:50.506643 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:50.506618    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 07 02:56:51.077450 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:51.077405    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:56:51.980657 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:51.980615    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d50fec74-db2b-4a4e-b2a1-f76ec69fd868" path="/var/lib/kubelet/pods/d50fec74-db2b-4a4e-b2a1-f76ec69fd868/volumes"
Jul 07 02:56:59.139103 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.139022    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"]
Jul 07 02:56:59.139511 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.139419    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d50fec74-db2b-4a4e-b2a1-f76ec69fd868" containerName="main"
Jul 07 02:56:59.139511 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.139436    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d50fec74-db2b-4a4e-b2a1-f76ec69fd868" containerName="main"
Jul 07 02:56:59.139511 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.139456    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main"
Jul 07 02:56:59.139511 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.139461    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main"
Jul 07 02:56:59.139647 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.139533    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d50fec74-db2b-4a4e-b2a1-f76ec69fd868" containerName="main"
Jul 07 02:56:59.139647 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.139542    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="682f1053-c9ea-4561-8827-9fb7d405bfce" containerName="main"
Jul 07 02:56:59.142450 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.142433    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.145075 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.145053    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 07 02:56:59.156082 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.156055    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"]
Jul 07 02:56:59.215960 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.215928    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-kserve-provision-location\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.216110 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.215963    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-home\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.216110 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.215985    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4180bdd-0598-4ae1-b996-97da966af13e-tls-certs\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.216110 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.216060    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7rsj\" (UniqueName: \"kubernetes.io/projected/b4180bdd-0598-4ae1-b996-97da966af13e-kube-api-access-l7rsj\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.216110 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.216093    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-model-cache\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.216281 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.216121    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-tmp-dir\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.216281 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.216208    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-dshm\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.316748 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.316703    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-model-cache\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.316941 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.316768    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-tmp-dir\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.316941 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.316833    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-dshm\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.316941 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.316901    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-kserve-provision-location\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.316941 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.316933    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-home\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.317167 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.316963    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4180bdd-0598-4ae1-b996-97da966af13e-tls-certs\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.317167 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.317002    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7rsj\" (UniqueName: \"kubernetes.io/projected/b4180bdd-0598-4ae1-b996-97da966af13e-kube-api-access-l7rsj\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.317167 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.317153    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-model-cache\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.317381 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.317357    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-home\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.317557 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.317533    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-tmp-dir\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.317642 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.317576    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-kserve-provision-location\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.319680 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.319651    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4180bdd-0598-4ae1-b996-97da966af13e-tls-certs\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.319817 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.319793    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-dshm\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.325487 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.325467    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7rsj\" (UniqueName: \"kubernetes.io/projected/b4180bdd-0598-4ae1-b996-97da966af13e-kube-api-access-l7rsj\") pod \"stop-feature-test-kserve-78ddd7447-7jzbj\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.452262 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.452156    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:56:59.607381 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:56:59.607348    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"]
Jul 07 02:56:59.610870 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:56:59.610841    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4180bdd_0598_4ae1_b996_97da966af13e.slice/crio-40ce42829a5de281e214493d57c568ccdf2f0f4e716828d348f70967fa94db6f WatchSource:0}: Error finding container 40ce42829a5de281e214493d57c568ccdf2f0f4e716828d348f70967fa94db6f: Status 404 returned error can't find the container with id 40ce42829a5de281e214493d57c568ccdf2f0f4e716828d348f70967fa94db6f
Jul 07 02:57:00.534386 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:00.534328    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" event={"ID":"b4180bdd-0598-4ae1-b996-97da966af13e","Type":"ContainerStarted","Data":"34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647"}
Jul 07 02:57:00.535367 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:00.534512    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" event={"ID":"b4180bdd-0598-4ae1-b996-97da966af13e","Type":"ContainerStarted","Data":"40ce42829a5de281e214493d57c568ccdf2f0f4e716828d348f70967fa94db6f"}
Jul 07 02:57:01.078256 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:01.078213    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:57:03.563967 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:03.563936    2543 generic.go:356] "Generic (PLEG): container finished" podID="b4180bdd-0598-4ae1-b996-97da966af13e" containerID="34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647" exitCode=0
Jul 07 02:57:03.564430 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:03.564004    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" event={"ID":"b4180bdd-0598-4ae1-b996-97da966af13e","Type":"ContainerDied","Data":"34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647"}
Jul 07 02:57:04.572205 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:04.572148    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" event={"ID":"b4180bdd-0598-4ae1-b996-97da966af13e","Type":"ContainerStarted","Data":"456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e"}
Jul 07 02:57:04.595853 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:04.595788    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podStartSLOduration=2.275683421 podStartE2EDuration="5.595764299s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.320080878s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:57:04.594086392 +0000 UTC m=+2337.237246915" watchObservedRunningTime="2026-07-07 02:57:04.595764299 +0000 UTC m=+2337.238924823"
Jul 07 02:57:05.452707 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:05.452670    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:57:09.452583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:09.452551    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:57:09.454602 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:09.454577    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:57:11.078432 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:11.078386    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:57:19.453509 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:19.453468    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:57:21.078283 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:21.078242    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:57:29.453241 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:29.453175    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:57:31.077487 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:31.077442    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:57:39.453116 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:39.453065    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:57:41.078168 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:41.078119    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:57:49.453455 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:49.453410    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:57:51.077578 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:51.077533    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:57:59.452605 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:57:59.452558    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:58:01.087301 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:01.087266    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:58:01.094998 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:01.094979    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:58:05.678502 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:05.678465    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"]
Jul 07 02:58:08.012176 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:08.012147    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:58:08.017882 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:08.017856    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 02:58:09.453583 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:09.453540    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:58:19.453400 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:19.453353    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:58:20.702814 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:20.702774    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" containerID="cri-o://4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46" gracePeriod=45
Jul 07 02:58:21.088975 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.088883    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": dial tcp 10.134.0.48:8000: connect: connection refused"
Jul 07 02:58:21.764514 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.764488    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:58:21.794844 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.794775    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-kserve-provision-location\") pod \"548a3b24-ae47-47eb-86df-7807ec615923\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") "
Jul 07 02:58:21.794844 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.794829    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/548a3b24-ae47-47eb-86df-7807ec615923-tls-certs\") pod \"548a3b24-ae47-47eb-86df-7807ec615923\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") "
Jul 07 02:58:21.795056 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.794873    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-home\") pod \"548a3b24-ae47-47eb-86df-7807ec615923\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") "
Jul 07 02:58:21.795056 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.794896    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-dshm\") pod \"548a3b24-ae47-47eb-86df-7807ec615923\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") "
Jul 07 02:58:21.795056 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.794969    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-model-cache\") pod \"548a3b24-ae47-47eb-86df-7807ec615923\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") "
Jul 07 02:58:21.795056 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.794998    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbrpz\" (UniqueName: \"kubernetes.io/projected/548a3b24-ae47-47eb-86df-7807ec615923-kube-api-access-wbrpz\") pod \"548a3b24-ae47-47eb-86df-7807ec615923\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") "
Jul 07 02:58:21.795056 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.795029    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-tmp-dir\") pod \"548a3b24-ae47-47eb-86df-7807ec615923\" (UID: \"548a3b24-ae47-47eb-86df-7807ec615923\") "
Jul 07 02:58:21.795769 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.795414    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-model-cache" (OuterVolumeSpecName: "model-cache") pod "548a3b24-ae47-47eb-86df-7807ec615923" (UID: "548a3b24-ae47-47eb-86df-7807ec615923"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:58:21.795769 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.795732    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-home" (OuterVolumeSpecName: "home") pod "548a3b24-ae47-47eb-86df-7807ec615923" (UID: "548a3b24-ae47-47eb-86df-7807ec615923"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:58:21.797827 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.797787    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548a3b24-ae47-47eb-86df-7807ec615923-kube-api-access-wbrpz" (OuterVolumeSpecName: "kube-api-access-wbrpz") pod "548a3b24-ae47-47eb-86df-7807ec615923" (UID: "548a3b24-ae47-47eb-86df-7807ec615923"). InnerVolumeSpecName "kube-api-access-wbrpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:58:21.798083 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.798059    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-dshm" (OuterVolumeSpecName: "dshm") pod "548a3b24-ae47-47eb-86df-7807ec615923" (UID: "548a3b24-ae47-47eb-86df-7807ec615923"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:58:21.798168 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.798105    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548a3b24-ae47-47eb-86df-7807ec615923-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "548a3b24-ae47-47eb-86df-7807ec615923" (UID: "548a3b24-ae47-47eb-86df-7807ec615923"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:58:21.813899 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.813857    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "548a3b24-ae47-47eb-86df-7807ec615923" (UID: "548a3b24-ae47-47eb-86df-7807ec615923"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:58:21.857698 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.857660    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "548a3b24-ae47-47eb-86df-7807ec615923" (UID: "548a3b24-ae47-47eb-86df-7807ec615923"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:58:21.896612 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.896581    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:58:21.896612 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.896611    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbrpz\" (UniqueName: \"kubernetes.io/projected/548a3b24-ae47-47eb-86df-7807ec615923-kube-api-access-wbrpz\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:58:21.896788 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.896620    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:58:21.896788 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.896635    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:58:21.896788 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.896646    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/548a3b24-ae47-47eb-86df-7807ec615923-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:58:21.896788 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.896656    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:58:21.896788 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.896663    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/548a3b24-ae47-47eb-86df-7807ec615923-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:58:21.973473 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.973441    2543 generic.go:356] "Generic (PLEG): container finished" podID="548a3b24-ae47-47eb-86df-7807ec615923" containerID="4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46" exitCode=0
Jul 07 02:58:21.973642 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.973497    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"
Jul 07 02:58:21.973642 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.973504    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" event={"ID":"548a3b24-ae47-47eb-86df-7807ec615923","Type":"ContainerDied","Data":"4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46"}
Jul 07 02:58:21.973642 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.973554    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf" event={"ID":"548a3b24-ae47-47eb-86df-7807ec615923","Type":"ContainerDied","Data":"1c58c6cf8db6ac0f05533ba5f78fa20144b02689f91229741a105d0837864037"}
Jul 07 02:58:21.973642 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.973582    2543 scope.go:117] "RemoveContainer" containerID="4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46"
Jul 07 02:58:21.983778 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.983753    2543 scope.go:117] "RemoveContainer" containerID="5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830"
Jul 07 02:58:21.993711 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.993690    2543 scope.go:117] "RemoveContainer" containerID="4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46"
Jul 07 02:58:21.993998 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:58:21.993979    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46\": container with ID starting with 4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46 not found: ID does not exist" containerID="4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46"
Jul 07 02:58:21.994064 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.994007    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46"} err="failed to get container status \"4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46\": rpc error: code = NotFound desc = could not find container \"4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46\": container with ID starting with 4e0b8b15641dfe3a9ad97c9faef7343fc9793470e750e90b8287074dd40d7e46 not found: ID does not exist"
Jul 07 02:58:21.994064 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.994023    2543 scope.go:117] "RemoveContainer" containerID="5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830"
Jul 07 02:58:21.994303 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:58:21.994283    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830\": container with ID starting with 5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830 not found: ID does not exist" containerID="5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830"
Jul 07 02:58:21.994356 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:21.994310    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830"} err="failed to get container status \"5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830\": rpc error: code = NotFound desc = could not find container \"5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830\": container with ID starting with 5382038f15aca65b1a842eb46bd7b63040b4a6f8e13998a28eeaaed5e6745830 not found: ID does not exist"
Jul 07 02:58:22.000025 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:22.000002    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"]
Jul 07 02:58:22.004705 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:22.004684    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-ffb76d85c-n2jtf"]
Jul 07 02:58:23.986811 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:23.981917    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="548a3b24-ae47-47eb-86df-7807ec615923" path="/var/lib/kubelet/pods/548a3b24-ae47-47eb-86df-7807ec615923/volumes"
Jul 07 02:58:29.453019 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:29.452925    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:58:32.040797 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.040756    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"]
Jul 07 02:58:32.041297 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.041276    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="storage-initializer"
Jul 07 02:58:32.041349 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.041302    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="storage-initializer"
Jul 07 02:58:32.041349 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.041327    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main"
Jul 07 02:58:32.041349 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.041336    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main"
Jul 07 02:58:32.041474 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.041460    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="548a3b24-ae47-47eb-86df-7807ec615923" containerName="main"
Jul 07 02:58:32.044995 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.044974    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.047595 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.047569    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 07 02:58:32.054862 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.054838    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"]
Jul 07 02:58:32.087130 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.087101    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-dshm\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.087359 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.087333    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-model-cache\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.087512 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.087489    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.087595 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.087527    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwf5t\" (UniqueName: \"kubernetes.io/projected/2c8a42f1-35a0-4081-98fb-b71af5126c90-kube-api-access-fwf5t\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.087595 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.087559    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-tmp-dir\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.087812 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.087793    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-home\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.087867 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.087824    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8a42f1-35a0-4081-98fb-b71af5126c90-tls-certs\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.188646 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.188605    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-tmp-dir\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.188848 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.188659    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-home\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.188848 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.188684    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8a42f1-35a0-4081-98fb-b71af5126c90-tls-certs\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.188848 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.188728    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-dshm\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.188848 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.188788    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-model-cache\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.188848 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.188824    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.189112 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.188847    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwf5t\" (UniqueName: \"kubernetes.io/projected/2c8a42f1-35a0-4081-98fb-b71af5126c90-kube-api-access-fwf5t\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.189112 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.189092    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-home\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.189229 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.189153    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-tmp-dir\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.189345 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.189324    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-model-cache\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.189418 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.189397    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.191335 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.191308    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-dshm\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.191560 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.191539    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8a42f1-35a0-4081-98fb-b71af5126c90-tls-certs\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.197869 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.197841    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwf5t\" (UniqueName: \"kubernetes.io/projected/2c8a42f1-35a0-4081-98fb-b71af5126c90-kube-api-access-fwf5t\") pod \"custom-route-timeout-test-kserve-66c886969f-7v7hs\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.357901 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.357822    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:32.503563 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:32.503537    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"]
Jul 07 02:58:32.507053 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:58:32.507022    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8a42f1_35a0_4081_98fb_b71af5126c90.slice/crio-093d18c3ee4719738e778bd33fc8d31e816dd6abae738d21ea6abd1065aa8bde WatchSource:0}: Error finding container 093d18c3ee4719738e778bd33fc8d31e816dd6abae738d21ea6abd1065aa8bde: Status 404 returned error can't find the container with id 093d18c3ee4719738e778bd33fc8d31e816dd6abae738d21ea6abd1065aa8bde
Jul 07 02:58:33.035478 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:33.035440    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" event={"ID":"2c8a42f1-35a0-4081-98fb-b71af5126c90","Type":"ContainerStarted","Data":"6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01"}
Jul 07 02:58:33.035683 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:33.035492    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" event={"ID":"2c8a42f1-35a0-4081-98fb-b71af5126c90","Type":"ContainerStarted","Data":"093d18c3ee4719738e778bd33fc8d31e816dd6abae738d21ea6abd1065aa8bde"}
Jul 07 02:58:36.052858 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:36.052827    2543 generic.go:356] "Generic (PLEG): container finished" podID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerID="6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01" exitCode=0
Jul 07 02:58:36.053335 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:36.052859    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" event={"ID":"2c8a42f1-35a0-4081-98fb-b71af5126c90","Type":"ContainerDied","Data":"6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01"}
Jul 07 02:58:37.059771 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:37.059728    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" event={"ID":"2c8a42f1-35a0-4081-98fb-b71af5126c90","Type":"ContainerStarted","Data":"53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9"}
Jul 07 02:58:37.082791 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:37.082726    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podStartSLOduration=1.802320226 podStartE2EDuration="5.082705273s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.280385047s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:58:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:58:37.080402368 +0000 UTC m=+2429.723562890" watchObservedRunningTime="2026-07-07 02:58:37.082705273 +0000 UTC m=+2429.725865796"
Jul 07 02:58:37.358897 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:37.358808    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:39.452808 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:39.452757    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:58:42.358880 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:42.358838    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 02:58:42.360766 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:42.360729    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 02:58:49.453356 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:49.453301    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:58:52.358530 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:52.358473    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 02:58:59.462794 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:59.462751    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:58:59.470747 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:58:59.470718    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:59:00.723352 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:00.723310    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"]
Jul 07 02:59:02.358774 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:02.358716    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 02:59:12.359003 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:12.358949    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 02:59:16.205648 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:16.205574    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" containerID="cri-o://456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e" gracePeriod=45
Jul 07 02:59:16.464054 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:16.463962    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": dial tcp 10.134.0.49:8000: connect: connection refused"
Jul 07 02:59:16.979119 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:16.979093    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:59:17.114543 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.114507    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4180bdd-0598-4ae1-b996-97da966af13e-tls-certs\") pod \"b4180bdd-0598-4ae1-b996-97da966af13e\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") "
Jul 07 02:59:17.114543 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.114547    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-tmp-dir\") pod \"b4180bdd-0598-4ae1-b996-97da966af13e\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") "
Jul 07 02:59:17.114796 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.114583    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l7rsj\" (UniqueName: \"kubernetes.io/projected/b4180bdd-0598-4ae1-b996-97da966af13e-kube-api-access-l7rsj\") pod \"b4180bdd-0598-4ae1-b996-97da966af13e\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") "
Jul 07 02:59:17.114796 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.114605    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-model-cache\") pod \"b4180bdd-0598-4ae1-b996-97da966af13e\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") "
Jul 07 02:59:17.114796 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.114639    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-kserve-provision-location\") pod \"b4180bdd-0598-4ae1-b996-97da966af13e\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") "
Jul 07 02:59:17.114796 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.114682    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-dshm\") pod \"b4180bdd-0598-4ae1-b996-97da966af13e\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") "
Jul 07 02:59:17.114796 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.114721    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-home\") pod \"b4180bdd-0598-4ae1-b996-97da966af13e\" (UID: \"b4180bdd-0598-4ae1-b996-97da966af13e\") "
Jul 07 02:59:17.115062 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.114826    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4180bdd-0598-4ae1-b996-97da966af13e" (UID: "b4180bdd-0598-4ae1-b996-97da966af13e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:59:17.115123 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.115082    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:59:17.115474 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.115451    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-home" (OuterVolumeSpecName: "home") pod "b4180bdd-0598-4ae1-b996-97da966af13e" (UID: "b4180bdd-0598-4ae1-b996-97da966af13e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:59:17.117347 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.117316    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4180bdd-0598-4ae1-b996-97da966af13e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4180bdd-0598-4ae1-b996-97da966af13e" (UID: "b4180bdd-0598-4ae1-b996-97da966af13e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 02:59:17.117474 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.117258    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4180bdd-0598-4ae1-b996-97da966af13e-kube-api-access-l7rsj" (OuterVolumeSpecName: "kube-api-access-l7rsj") pod "b4180bdd-0598-4ae1-b996-97da966af13e" (UID: "b4180bdd-0598-4ae1-b996-97da966af13e"). InnerVolumeSpecName "kube-api-access-l7rsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 02:59:17.118566 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.118537    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-dshm" (OuterVolumeSpecName: "dshm") pod "b4180bdd-0598-4ae1-b996-97da966af13e" (UID: "b4180bdd-0598-4ae1-b996-97da966af13e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:59:17.132647 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.132615    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4180bdd-0598-4ae1-b996-97da966af13e" (UID: "b4180bdd-0598-4ae1-b996-97da966af13e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:59:17.185586 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.185541    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4180bdd-0598-4ae1-b996-97da966af13e" (UID: "b4180bdd-0598-4ae1-b996-97da966af13e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 02:59:17.216037 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.216006    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4180bdd-0598-4ae1-b996-97da966af13e-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:59:17.216037 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.216032    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:59:17.216037 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.216042    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7rsj\" (UniqueName: \"kubernetes.io/projected/b4180bdd-0598-4ae1-b996-97da966af13e-kube-api-access-l7rsj\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:59:17.216485 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.216053    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:59:17.216485 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.216062    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:59:17.216485 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.216072    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4180bdd-0598-4ae1-b996-97da966af13e-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 02:59:17.264472 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.264440    2543 generic.go:356] "Generic (PLEG): container finished" podID="b4180bdd-0598-4ae1-b996-97da966af13e" containerID="456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e" exitCode=0
Jul 07 02:59:17.264663 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.264515    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"
Jul 07 02:59:17.264663 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.264536    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" event={"ID":"b4180bdd-0598-4ae1-b996-97da966af13e","Type":"ContainerDied","Data":"456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e"}
Jul 07 02:59:17.264663 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.264613    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj" event={"ID":"b4180bdd-0598-4ae1-b996-97da966af13e","Type":"ContainerDied","Data":"40ce42829a5de281e214493d57c568ccdf2f0f4e716828d348f70967fa94db6f"}
Jul 07 02:59:17.264663 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.264647    2543 scope.go:117] "RemoveContainer" containerID="456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e"
Jul 07 02:59:17.275654 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.275608    2543 scope.go:117] "RemoveContainer" containerID="34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647"
Jul 07 02:59:17.284294 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.284270    2543 scope.go:117] "RemoveContainer" containerID="456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e"
Jul 07 02:59:17.284837 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:59:17.284708    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e\": container with ID starting with 456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e not found: ID does not exist" containerID="456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e"
Jul 07 02:59:17.284837 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.284749    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e"} err="failed to get container status \"456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e\": rpc error: code = NotFound desc = could not find container \"456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e\": container with ID starting with 456573f6d0e8e54b813b5771f7594dac88472489d40fd01b8131222c8de2973e not found: ID does not exist"
Jul 07 02:59:17.284837 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.284788    2543 scope.go:117] "RemoveContainer" containerID="34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647"
Jul 07 02:59:17.285136 ip-10-0-132-101 kubenswrapper[2543]: E0707 02:59:17.285102    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647\": container with ID starting with 34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647 not found: ID does not exist" containerID="34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647"
Jul 07 02:59:17.285219 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.285144    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647"} err="failed to get container status \"34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647\": rpc error: code = NotFound desc = could not find container \"34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647\": container with ID starting with 34baa55bd7b9942045242c63c320b617211f324594af4e2c7cc16eaa4f0f9647 not found: ID does not exist"
Jul 07 02:59:17.286996 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.286977    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"]
Jul 07 02:59:17.290829 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.290807    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-7jzbj"]
Jul 07 02:59:17.980813 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:17.980776    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" path="/var/lib/kubelet/pods/b4180bdd-0598-4ae1-b996-97da966af13e/volumes"
Jul 07 02:59:18.553926 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.553884    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"]
Jul 07 02:59:18.554405 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.554393    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main"
Jul 07 02:59:18.554479 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.554411    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main"
Jul 07 02:59:18.554479 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.554449    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="storage-initializer"
Jul 07 02:59:18.554479 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.554459    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="storage-initializer"
Jul 07 02:59:18.554644 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.554537    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4180bdd-0598-4ae1-b996-97da966af13e" containerName="main"
Jul 07 02:59:18.566425 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.566393    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.569074 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.569045    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"]
Jul 07 02:59:18.569355 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.569338    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 07 02:59:18.626972 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.626931    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-model-cache\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.627169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.626978    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-dshm\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.627169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.627017    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tmp-dir\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.627169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.627073    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-home\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.627169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.627107    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tls-certs\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.627169 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.627159    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kserve-provision-location\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.627414 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.627235    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbl2\" (UniqueName: \"kubernetes.io/projected/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kube-api-access-dcbl2\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.728551 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.728513    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kserve-provision-location\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.728750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.728562    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbl2\" (UniqueName: \"kubernetes.io/projected/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kube-api-access-dcbl2\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.728750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.728589    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-model-cache\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.728750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.728611    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-dshm\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.728750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.728635    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tmp-dir\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.728750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.728662    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-home\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.728750 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.728695    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tls-certs\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.729079 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.729011    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kserve-provision-location\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.729079 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.729042    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-model-cache\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.729156 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.729089    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tmp-dir\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.729259 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.729242    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-home\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.730974 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.730949    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-dshm\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.731720 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.731691    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tls-certs\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.737134 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.737111    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbl2\" (UniqueName: \"kubernetes.io/projected/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kube-api-access-dcbl2\") pod \"stop-feature-test-kserve-78ddd7447-8kcjs\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:18.878851 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:18.878822    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:19.030635 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:19.030599    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"]
Jul 07 02:59:19.034274 ip-10-0-132-101 kubenswrapper[2543]: W0707 02:59:19.034240    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06e43a9e_6d18_4618_97c0_8ccbf44335fb.slice/crio-644d9eaf74e68461ed28f0a57c9cab6dcd3a99dd9d038fc11e8c5120bd35bca2 WatchSource:0}: Error finding container 644d9eaf74e68461ed28f0a57c9cab6dcd3a99dd9d038fc11e8c5120bd35bca2: Status 404 returned error can't find the container with id 644d9eaf74e68461ed28f0a57c9cab6dcd3a99dd9d038fc11e8c5120bd35bca2
Jul 07 02:59:19.278459 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:19.278361    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" event={"ID":"06e43a9e-6d18-4618-97c0-8ccbf44335fb","Type":"ContainerStarted","Data":"136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27"}
Jul 07 02:59:19.278459 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:19.278427    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" event={"ID":"06e43a9e-6d18-4618-97c0-8ccbf44335fb","Type":"ContainerStarted","Data":"644d9eaf74e68461ed28f0a57c9cab6dcd3a99dd9d038fc11e8c5120bd35bca2"}
Jul 07 02:59:22.294464 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:22.294429    2543 generic.go:356] "Generic (PLEG): container finished" podID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerID="136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27" exitCode=0
Jul 07 02:59:22.294937 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:22.294487    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" event={"ID":"06e43a9e-6d18-4618-97c0-8ccbf44335fb","Type":"ContainerDied","Data":"136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27"}
Jul 07 02:59:22.359072 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:22.359041    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 02:59:23.302172 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:23.302138    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" event={"ID":"06e43a9e-6d18-4618-97c0-8ccbf44335fb","Type":"ContainerStarted","Data":"566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc"}
Jul 07 02:59:23.325482 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:23.325427    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podStartSLOduration=2.293654402 podStartE2EDuration="5.325409709s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.031755307s" isStatefulPod=false podCreationTimestamp="2026-07-07 02:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 02:59:23.322492387 +0000 UTC m=+2475.965652946" watchObservedRunningTime="2026-07-07 02:59:23.325409709 +0000 UTC m=+2475.968570283"
Jul 07 02:59:23.879042 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:23.879003    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:28.879831 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:28.879795    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 02:59:28.881700 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:28.881668    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 02:59:32.359037 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:32.358990    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 02:59:38.879650 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:38.879599    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 02:59:42.359433 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:42.359383    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 02:59:48.879713 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:48.879667    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 02:59:52.358702 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:52.358647    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 02:59:58.879612 ip-10-0-132-101 kubenswrapper[2543]: I0707 02:59:58.879508    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 03:00:02.358911 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:02.358856    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 03:00:08.879845 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:08.879797    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 03:00:12.359074 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:12.359033    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 03:00:18.879801 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:18.879759    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 03:00:22.358781 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:22.358737    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 07 03:00:28.880413 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:28.880359    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 03:00:32.368866 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:32.368831    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 03:00:32.376842 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:32.376817    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 03:00:38.879365 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:38.879315    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 03:00:39.343268 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:39.343159    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"]
Jul 07 03:00:48.879920 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:48.879876    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 03:00:54.368352 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:54.368286    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main" containerID="cri-o://53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9" gracePeriod=45
Jul 07 03:00:55.346883 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.346851    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 03:00:55.439774 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.439691    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8a42f1-35a0-4081-98fb-b71af5126c90-tls-certs\") pod \"2c8a42f1-35a0-4081-98fb-b71af5126c90\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") "
Jul 07 03:00:55.439774 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.439730    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-tmp-dir\") pod \"2c8a42f1-35a0-4081-98fb-b71af5126c90\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") "
Jul 07 03:00:55.440339 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.439780    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-kserve-provision-location\") pod \"2c8a42f1-35a0-4081-98fb-b71af5126c90\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") "
Jul 07 03:00:55.440339 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.439842    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwf5t\" (UniqueName: \"kubernetes.io/projected/2c8a42f1-35a0-4081-98fb-b71af5126c90-kube-api-access-fwf5t\") pod \"2c8a42f1-35a0-4081-98fb-b71af5126c90\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") "
Jul 07 03:00:55.440339 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.439893    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-home\") pod \"2c8a42f1-35a0-4081-98fb-b71af5126c90\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") "
Jul 07 03:00:55.440339 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.439912    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-dshm\") pod \"2c8a42f1-35a0-4081-98fb-b71af5126c90\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") "
Jul 07 03:00:55.440339 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.439933    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-model-cache\") pod \"2c8a42f1-35a0-4081-98fb-b71af5126c90\" (UID: \"2c8a42f1-35a0-4081-98fb-b71af5126c90\") "
Jul 07 03:00:55.440590 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.440404    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c8a42f1-35a0-4081-98fb-b71af5126c90" (UID: "2c8a42f1-35a0-4081-98fb-b71af5126c90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:00:55.440590 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.440569    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-home" (OuterVolumeSpecName: "home") pod "2c8a42f1-35a0-4081-98fb-b71af5126c90" (UID: "2c8a42f1-35a0-4081-98fb-b71af5126c90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:00:55.442489 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.442425    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8a42f1-35a0-4081-98fb-b71af5126c90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c8a42f1-35a0-4081-98fb-b71af5126c90" (UID: "2c8a42f1-35a0-4081-98fb-b71af5126c90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:00:55.442770 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.442733    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8a42f1-35a0-4081-98fb-b71af5126c90-kube-api-access-fwf5t" (OuterVolumeSpecName: "kube-api-access-fwf5t") pod "2c8a42f1-35a0-4081-98fb-b71af5126c90" (UID: "2c8a42f1-35a0-4081-98fb-b71af5126c90"). InnerVolumeSpecName "kube-api-access-fwf5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:00:55.442879 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.442813    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-dshm" (OuterVolumeSpecName: "dshm") pod "2c8a42f1-35a0-4081-98fb-b71af5126c90" (UID: "2c8a42f1-35a0-4081-98fb-b71af5126c90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:00:55.459110 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.459071    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c8a42f1-35a0-4081-98fb-b71af5126c90" (UID: "2c8a42f1-35a0-4081-98fb-b71af5126c90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:00:55.501600 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.501560    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c8a42f1-35a0-4081-98fb-b71af5126c90" (UID: "2c8a42f1-35a0-4081-98fb-b71af5126c90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:00:55.541605 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.541562    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8a42f1-35a0-4081-98fb-b71af5126c90-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:00:55.541605 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.541602    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:00:55.541782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.541619    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:00:55.541782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.541634    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwf5t\" (UniqueName: \"kubernetes.io/projected/2c8a42f1-35a0-4081-98fb-b71af5126c90-kube-api-access-fwf5t\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:00:55.541782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.541647    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:00:55.541782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.541662    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:00:55.541782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.541674    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c8a42f1-35a0-4081-98fb-b71af5126c90-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:00:55.786100 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.786010    2543 generic.go:356] "Generic (PLEG): container finished" podID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerID="53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9" exitCode=0
Jul 07 03:00:55.786100 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.786061    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"
Jul 07 03:00:55.786100 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.786073    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" event={"ID":"2c8a42f1-35a0-4081-98fb-b71af5126c90","Type":"ContainerDied","Data":"53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9"}
Jul 07 03:00:55.786347 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.786133    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs" event={"ID":"2c8a42f1-35a0-4081-98fb-b71af5126c90","Type":"ContainerDied","Data":"093d18c3ee4719738e778bd33fc8d31e816dd6abae738d21ea6abd1065aa8bde"}
Jul 07 03:00:55.786347 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.786163    2543 scope.go:117] "RemoveContainer" containerID="53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9"
Jul 07 03:00:55.795400 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.795382    2543 scope.go:117] "RemoveContainer" containerID="6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01"
Jul 07 03:00:55.804300 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.804282    2543 scope.go:117] "RemoveContainer" containerID="53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9"
Jul 07 03:00:55.804598 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:00:55.804579    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9\": container with ID starting with 53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9 not found: ID does not exist" containerID="53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9"
Jul 07 03:00:55.804646 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.804605    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9"} err="failed to get container status \"53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9\": rpc error: code = NotFound desc = could not find container \"53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9\": container with ID starting with 53c92f94fe29fa54e9f5401367566fb092bee51b4ed7064b449e854c4842cfa9 not found: ID does not exist"
Jul 07 03:00:55.804646 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.804622    2543 scope.go:117] "RemoveContainer" containerID="6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01"
Jul 07 03:00:55.804863 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:00:55.804847    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01\": container with ID starting with 6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01 not found: ID does not exist" containerID="6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01"
Jul 07 03:00:55.804916 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.804865    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01"} err="failed to get container status \"6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01\": rpc error: code = NotFound desc = could not find container \"6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01\": container with ID starting with 6b516b6b00e55b31de4edee3bf39613d39d477b2d18d7d31114c50c952da1a01 not found: ID does not exist"
Jul 07 03:00:55.808857 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.808833    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"]
Jul 07 03:00:55.812987 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.812966    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-66c886969f-7v7hs"]
Jul 07 03:00:55.981305 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:55.981269    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" path="/var/lib/kubelet/pods/2c8a42f1-35a0-4081-98fb-b71af5126c90/volumes"
Jul 07 03:00:58.879811 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:00:58.879761    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 03:01:01.577752 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:01.577715    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"]
Jul 07 03:01:01.578231 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:01.577944    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy" containerID="cri-o://86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94" gracePeriod=30
Jul 07 03:01:06.838647 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.838622    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 03:01:06.842378 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.842321    2543 generic.go:356] "Generic (PLEG): container finished" podID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerID="86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94" exitCode=0
Jul 07 03:01:06.842378 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.842366    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"
Jul 07 03:01:06.842555 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.842388    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" event={"ID":"61609f51-bd34-49c2-89b2-b7ee16ba35c4","Type":"ContainerDied","Data":"86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94"}
Jul 07 03:01:06.842555 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.842430    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l" event={"ID":"61609f51-bd34-49c2-89b2-b7ee16ba35c4","Type":"ContainerDied","Data":"38d5f78842096861c990be66a6f41c4b570c048d5eea2851a1048d151ffa680a"}
Jul 07 03:01:06.842555 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.842452    2543 scope.go:117] "RemoveContainer" containerID="86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94"
Jul 07 03:01:06.850063 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850002    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-data\") pod \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") "
Jul 07 03:01:06.850215 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850075    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-token\") pod \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") "
Jul 07 03:01:06.850215 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850102    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm7x5\" (UniqueName: \"kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-kube-api-access-nm7x5\") pod \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") "
Jul 07 03:01:06.850215 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850122    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-podinfo\") pod \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") "
Jul 07 03:01:06.850215 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850150    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-credential-socket\") pod \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") "
Jul 07 03:01:06.850450 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850225    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-socket\") pod \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") "
Jul 07 03:01:06.850450 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850280    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-envoy\") pod \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") "
Jul 07 03:01:06.850450 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850313    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istiod-ca-cert\") pod \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") "
Jul 07 03:01:06.850450 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850366    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-certs\") pod \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\" (UID: \"61609f51-bd34-49c2-89b2-b7ee16ba35c4\") "
Jul 07 03:01:06.850671 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850274    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "61609f51-bd34-49c2-89b2-b7ee16ba35c4" (UID: "61609f51-bd34-49c2-89b2-b7ee16ba35c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:06.850671 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.850656    2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-data\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:06.851424 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.851040    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "61609f51-bd34-49c2-89b2-b7ee16ba35c4" (UID: "61609f51-bd34-49c2-89b2-b7ee16ba35c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:06.851424 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.851245    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "61609f51-bd34-49c2-89b2-b7ee16ba35c4" (UID: "61609f51-bd34-49c2-89b2-b7ee16ba35c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 03:01:06.851424 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.851326    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "61609f51-bd34-49c2-89b2-b7ee16ba35c4" (UID: "61609f51-bd34-49c2-89b2-b7ee16ba35c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:06.851424 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.851394    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "61609f51-bd34-49c2-89b2-b7ee16ba35c4" (UID: "61609f51-bd34-49c2-89b2-b7ee16ba35c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:06.852164 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.852133    2543 scope.go:117] "RemoveContainer" containerID="465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b"
Jul 07 03:01:06.852834 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.852808    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "61609f51-bd34-49c2-89b2-b7ee16ba35c4" (UID: "61609f51-bd34-49c2-89b2-b7ee16ba35c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 07 03:01:06.853169 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.853134    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "61609f51-bd34-49c2-89b2-b7ee16ba35c4" (UID: "61609f51-bd34-49c2-89b2-b7ee16ba35c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:01:06.853548 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.853516    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-kube-api-access-nm7x5" (OuterVolumeSpecName: "kube-api-access-nm7x5") pod "61609f51-bd34-49c2-89b2-b7ee16ba35c4" (UID: "61609f51-bd34-49c2-89b2-b7ee16ba35c4"). InnerVolumeSpecName "kube-api-access-nm7x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:01:06.854175 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.854148    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "61609f51-bd34-49c2-89b2-b7ee16ba35c4" (UID: "61609f51-bd34-49c2-89b2-b7ee16ba35c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:06.879671 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.879651    2543 scope.go:117] "RemoveContainer" containerID="86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94"
Jul 07 03:01:06.880095 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:01:06.880073    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94\": container with ID starting with 86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94 not found: ID does not exist" containerID="86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94"
Jul 07 03:01:06.880158 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.880106    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94"} err="failed to get container status \"86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94\": rpc error: code = NotFound desc = could not find container \"86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94\": container with ID starting with 86db32d76d77b4b237337b007d8a1b3e43af9b968e2dd720ed8c277db101df94 not found: ID does not exist"
Jul 07 03:01:06.880158 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.880127    2543 scope.go:117] "RemoveContainer" containerID="465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b"
Jul 07 03:01:06.880510 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:01:06.880493    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b\": container with ID starting with 465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b not found: ID does not exist" containerID="465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b"
Jul 07 03:01:06.880590 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.880516    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b"} err="failed to get container status \"465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b\": rpc error: code = NotFound desc = could not find container \"465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b\": container with ID starting with 465082c0b7307bf679697ae61ad57a61b3790330b896cb0d32b079a16d503d3b not found: ID does not exist"
Jul 07 03:01:06.951409 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.951368    2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:06.951409 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.951407    2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-token\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:06.951614 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.951422    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm7x5\" (UniqueName: \"kubernetes.io/projected/61609f51-bd34-49c2-89b2-b7ee16ba35c4-kube-api-access-nm7x5\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:06.951614 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.951437    2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-podinfo\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:06.951614 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.951451    2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-credential-socket\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:06.951614 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.951465    2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-workload-socket\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:06.951614 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.951480    2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istio-envoy\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:06.951614 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:06.951494    2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61609f51-bd34-49c2-89b2-b7ee16ba35c4-istiod-ca-cert\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:07.164660 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:07.164631    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"]
Jul 07 03:01:07.168292 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:07.168268    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-6cn2l"]
Jul 07 03:01:07.979712 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:07.979675    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" path="/var/lib/kubelet/pods/61609f51-bd34-49c2-89b2-b7ee16ba35c4/volumes"
Jul 07 03:01:08.746002 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.745965    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"]
Jul 07 03:01:08.746393 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746376    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main"
Jul 07 03:01:08.746461 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746399    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main"
Jul 07 03:01:08.746461 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746421    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy"
Jul 07 03:01:08.746461 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746431    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy"
Jul 07 03:01:08.746461 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746445    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="storage-initializer"
Jul 07 03:01:08.746461 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746452    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="storage-initializer"
Jul 07 03:01:08.746640 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746463    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy"
Jul 07 03:01:08.746640 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746470    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy"
Jul 07 03:01:08.746640 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746547    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy"
Jul 07 03:01:08.746640 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746554    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="61609f51-bd34-49c2-89b2-b7ee16ba35c4" containerName="istio-proxy"
Jul 07 03:01:08.746640 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.746564    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8a42f1-35a0-4081-98fb-b71af5126c90" containerName="main"
Jul 07 03:01:08.749670 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.749652    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.752108 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.752089    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 07 03:01:08.760210 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.760167    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"]
Jul 07 03:01:08.766517 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.766495    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-kserve-provision-location\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.766650 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.766527    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-dshm\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.766650 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.766570    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-home\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.766650 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.766586    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-tmp-dir\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.766833 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.766655    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2lk\" (UniqueName: \"kubernetes.io/projected/20b2d517-4974-4a51-be75-9d4aaa456b00-kube-api-access-xj2lk\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.766833 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.766693    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b2d517-4974-4a51-be75-9d4aaa456b00-tls-certs\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.766833 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.766719    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-model-cache\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.867979 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.867931    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-model-cache\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.868173 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.868000    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-kserve-provision-location\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.868173 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.868039    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-dshm\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.868173 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.868104    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-home\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.868351 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.868220    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-tmp-dir\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.868351 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.868261    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2lk\" (UniqueName: \"kubernetes.io/projected/20b2d517-4974-4a51-be75-9d4aaa456b00-kube-api-access-xj2lk\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.868351 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.868298    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b2d517-4974-4a51-be75-9d4aaa456b00-tls-certs\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.868521 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.868414    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-model-cache\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.868521 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.868425    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-kserve-provision-location\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.868521 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.868497    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-home\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.868689 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.868655    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-tmp-dir\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.870661 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.870638    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-dshm\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.870866 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.870847    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b2d517-4974-4a51-be75-9d4aaa456b00-tls-certs\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.877479 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.877462    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2lk\" (UniqueName: \"kubernetes.io/projected/20b2d517-4974-4a51-be75-9d4aaa456b00-kube-api-access-xj2lk\") pod \"router-with-refs-test-kserve-cd8b578dc-h8cml\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:08.879518 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:08.879480    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 03:01:09.063595 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:09.063507    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:09.209273 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:09.209179    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"]
Jul 07 03:01:09.213788 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:01:09.213752    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20b2d517_4974_4a51_be75_9d4aaa456b00.slice/crio-142aa501c2127339a63ebcc3a97d184006da428ec57c804f2490dd06940c12ca WatchSource:0}: Error finding container 142aa501c2127339a63ebcc3a97d184006da428ec57c804f2490dd06940c12ca: Status 404 returned error can't find the container with id 142aa501c2127339a63ebcc3a97d184006da428ec57c804f2490dd06940c12ca
Jul 07 03:01:09.858554 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:09.858518    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" event={"ID":"20b2d517-4974-4a51-be75-9d4aaa456b00","Type":"ContainerStarted","Data":"1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9"}
Jul 07 03:01:09.858756 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:09.858583    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" event={"ID":"20b2d517-4974-4a51-be75-9d4aaa456b00","Type":"ContainerStarted","Data":"142aa501c2127339a63ebcc3a97d184006da428ec57c804f2490dd06940c12ca"}
Jul 07 03:01:12.874909 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:12.874871    2543 generic.go:356] "Generic (PLEG): container finished" podID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerID="1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9" exitCode=0
Jul 07 03:01:12.874909 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:12.874913    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" event={"ID":"20b2d517-4974-4a51-be75-9d4aaa456b00","Type":"ContainerDied","Data":"1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9"}
Jul 07 03:01:13.882424 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:13.882380    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" event={"ID":"20b2d517-4974-4a51-be75-9d4aaa456b00","Type":"ContainerStarted","Data":"2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402"}
Jul 07 03:01:13.904783 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:13.904715    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podStartSLOduration=2.541618442 podStartE2EDuration="5.904696144s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.363077702s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:01:13.90307189 +0000 UTC m=+2586.546232412" watchObservedRunningTime="2026-07-07 03:01:13.904696144 +0000 UTC m=+2586.547856669"
Jul 07 03:01:14.064335 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:14.064290    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:18.879698 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:18.879646    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": dial tcp 10.134.0.51:8000: connect: connection refused"
Jul 07 03:01:19.064630 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:19.064582    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:01:19.066583 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:19.066550    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:01:28.890537 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:28.890451    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 03:01:28.899141 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:28.899115    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 03:01:29.064428 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:29.064388    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:01:30.098386 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:30.098349    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"]
Jul 07 03:01:39.064382 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:39.064338    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:01:45.122964 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:45.122891    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main" containerID="cri-o://566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc" gracePeriod=45
Jul 07 03:01:45.881657 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:45.881630    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 03:01:46.005266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.005124    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-model-cache\") pod \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") "
Jul 07 03:01:46.005266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.005220    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tls-certs\") pod \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") "
Jul 07 03:01:46.005266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.005254    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-dshm\") pod \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") "
Jul 07 03:01:46.005573 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.005292    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-home\") pod \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") "
Jul 07 03:01:46.005573 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.005339    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kserve-provision-location\") pod \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") "
Jul 07 03:01:46.005573 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.005377    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tmp-dir\") pod \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") "
Jul 07 03:01:46.005573 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.005400    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcbl2\" (UniqueName: \"kubernetes.io/projected/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kube-api-access-dcbl2\") pod \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\" (UID: \"06e43a9e-6d18-4618-97c0-8ccbf44335fb\") "
Jul 07 03:01:46.005573 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.005472    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "06e43a9e-6d18-4618-97c0-8ccbf44335fb" (UID: "06e43a9e-6d18-4618-97c0-8ccbf44335fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:46.005840 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.005687    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:46.006234 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.006180    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-home" (OuterVolumeSpecName: "home") pod "06e43a9e-6d18-4618-97c0-8ccbf44335fb" (UID: "06e43a9e-6d18-4618-97c0-8ccbf44335fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:46.007893 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.007868    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06e43a9e-6d18-4618-97c0-8ccbf44335fb" (UID: "06e43a9e-6d18-4618-97c0-8ccbf44335fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:01:46.008068 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.008042    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-dshm" (OuterVolumeSpecName: "dshm") pod "06e43a9e-6d18-4618-97c0-8ccbf44335fb" (UID: "06e43a9e-6d18-4618-97c0-8ccbf44335fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:46.008156 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.008134    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kube-api-access-dcbl2" (OuterVolumeSpecName: "kube-api-access-dcbl2") pod "06e43a9e-6d18-4618-97c0-8ccbf44335fb" (UID: "06e43a9e-6d18-4618-97c0-8ccbf44335fb"). InnerVolumeSpecName "kube-api-access-dcbl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:01:46.026867 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.026827    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06e43a9e-6d18-4618-97c0-8ccbf44335fb" (UID: "06e43a9e-6d18-4618-97c0-8ccbf44335fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:46.052804 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.052760    2543 generic.go:356] "Generic (PLEG): container finished" podID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerID="566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc" exitCode=0
Jul 07 03:01:46.052998 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.052809    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"
Jul 07 03:01:46.052998 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.052841    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" event={"ID":"06e43a9e-6d18-4618-97c0-8ccbf44335fb","Type":"ContainerDied","Data":"566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc"}
Jul 07 03:01:46.052998 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.052900    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs" event={"ID":"06e43a9e-6d18-4618-97c0-8ccbf44335fb","Type":"ContainerDied","Data":"644d9eaf74e68461ed28f0a57c9cab6dcd3a99dd9d038fc11e8c5120bd35bca2"}
Jul 07 03:01:46.052998 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.052927    2543 scope.go:117] "RemoveContainer" containerID="566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc"
Jul 07 03:01:46.062243 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.062216    2543 scope.go:117] "RemoveContainer" containerID="136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27"
Jul 07 03:01:46.071606 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.071587    2543 scope.go:117] "RemoveContainer" containerID="566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc"
Jul 07 03:01:46.071887 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:01:46.071863    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc\": container with ID starting with 566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc not found: ID does not exist" containerID="566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc"
Jul 07 03:01:46.071975 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.071903    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc"} err="failed to get container status \"566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc\": rpc error: code = NotFound desc = could not find container \"566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc\": container with ID starting with 566a1f04c7ce8ca91d039c62615152bd701b3196f9ee023f9588b6b87369abdc not found: ID does not exist"
Jul 07 03:01:46.071975 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.071927    2543 scope.go:117] "RemoveContainer" containerID="136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27"
Jul 07 03:01:46.072151 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:01:46.072131    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27\": container with ID starting with 136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27 not found: ID does not exist" containerID="136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27"
Jul 07 03:01:46.072249 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.072157    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27"} err="failed to get container status \"136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27\": rpc error: code = NotFound desc = could not find container \"136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27\": container with ID starting with 136698a44912bb41fb4f75d33338430d4ddca14edd7fcbbfc19c57bc7fb9fe27 not found: ID does not exist"
Jul 07 03:01:46.085267 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.085229    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "06e43a9e-6d18-4618-97c0-8ccbf44335fb" (UID: "06e43a9e-6d18-4618-97c0-8ccbf44335fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:01:46.106392 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.106353    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:46.106392 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.106377    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:46.106643 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.106452    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:46.106643 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.106461    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:46.106643 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.106471    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06e43a9e-6d18-4618-97c0-8ccbf44335fb-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:46.106643 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.106481    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcbl2\" (UniqueName: \"kubernetes.io/projected/06e43a9e-6d18-4618-97c0-8ccbf44335fb-kube-api-access-dcbl2\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:01:46.376523 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.376480    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"]
Jul 07 03:01:46.381132 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:46.381107    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-78ddd7447-8kcjs"]
Jul 07 03:01:47.980530 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:47.980490    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" path="/var/lib/kubelet/pods/06e43a9e-6d18-4618-97c0-8ccbf44335fb/volumes"
Jul 07 03:01:49.064208 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:49.064145    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:01:56.853446 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.853412    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"]
Jul 07 03:01:56.853971 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.853940    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="storage-initializer"
Jul 07 03:01:56.853971 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.853960    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="storage-initializer"
Jul 07 03:01:56.854100 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.853982    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main"
Jul 07 03:01:56.854100 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.853991    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main"
Jul 07 03:01:56.854100 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.854076    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="06e43a9e-6d18-4618-97c0-8ccbf44335fb" containerName="main"
Jul 07 03:01:56.858089 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.858067    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.860631 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.860609    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 07 03:01:56.860748 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.860610    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 03:01:56.870032 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.870008    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"]
Jul 07 03:01:56.896138 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.896103    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftjjh\" (UniqueName: \"kubernetes.io/projected/e71ac637-0424-4d41-b4ca-1577c77f6215-kube-api-access-ftjjh\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.896354 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.896154    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.896354 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.896237    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-model-cache\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.896354 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.896289    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-dshm\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.896354 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.896342    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-home\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.896569 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.896410    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-tmp-dir\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.896569 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.896458    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e71ac637-0424-4d41-b4ca-1577c77f6215-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.896569 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.896496    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71ac637-0424-4d41-b4ca-1577c77f6215-tls-certs\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997220 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997153    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftjjh\" (UniqueName: \"kubernetes.io/projected/e71ac637-0424-4d41-b4ca-1577c77f6215-kube-api-access-ftjjh\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997220 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997225    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997483 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997254    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-model-cache\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997483 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997279    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-dshm\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997483 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997322    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-home\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997483 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997354    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-tmp-dir\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997483 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997383    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e71ac637-0424-4d41-b4ca-1577c77f6215-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997483 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997412    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71ac637-0424-4d41-b4ca-1577c77f6215-tls-certs\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997808 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997674    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997808 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997704    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-model-cache\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.997808 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.997785    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-tmp-dir\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.998032 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.998009    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-home\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.998212 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.998160    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e71ac637-0424-4d41-b4ca-1577c77f6215-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.999805 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.999785    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-dshm\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:56.999964 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:56.999946    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71ac637-0424-4d41-b4ca-1577c77f6215-tls-certs\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:57.005919 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:57.005898    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftjjh\" (UniqueName: \"kubernetes.io/projected/e71ac637-0424-4d41-b4ca-1577c77f6215-kube-api-access-ftjjh\") pod \"lora-single-adapter-test-kserve-6f56f99c66-k5b57\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:57.169021 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:57.168983    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:01:57.329530 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:57.328945    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"]
Jul 07 03:01:57.334134 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:01:57.333848    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71ac637_0424_4d41_b4ca_1577c77f6215.slice/crio-46c8c1c049538667352f51e4353a4b6a92290120adeee0496b818d8b3da3eb32 WatchSource:0}: Error finding container 46c8c1c049538667352f51e4353a4b6a92290120adeee0496b818d8b3da3eb32: Status 404 returned error can't find the container with id 46c8c1c049538667352f51e4353a4b6a92290120adeee0496b818d8b3da3eb32
Jul 07 03:01:58.112765 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:58.112733    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" event={"ID":"e71ac637-0424-4d41-b4ca-1577c77f6215","Type":"ContainerStarted","Data":"a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445"}
Jul 07 03:01:58.113169 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:58.112783    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" event={"ID":"e71ac637-0424-4d41-b4ca-1577c77f6215","Type":"ContainerStarted","Data":"46c8c1c049538667352f51e4353a4b6a92290120adeee0496b818d8b3da3eb32"}
Jul 07 03:01:59.064920 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:01:59.064880    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:02:02.134258 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:02.134218    2543 generic.go:356] "Generic (PLEG): container finished" podID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerID="a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445" exitCode=0
Jul 07 03:02:02.134725 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:02.134285    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" event={"ID":"e71ac637-0424-4d41-b4ca-1577c77f6215","Type":"ContainerDied","Data":"a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445"}
Jul 07 03:02:03.142039 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:03.141994    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" event={"ID":"e71ac637-0424-4d41-b4ca-1577c77f6215","Type":"ContainerStarted","Data":"9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce"}
Jul 07 03:02:03.163563 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:03.163504    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podStartSLOduration=3.059925584 podStartE2EDuration="7.163480624s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.10355504s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:02:03.162157421 +0000 UTC m=+2635.805317947" watchObservedRunningTime="2026-07-07 03:02:03.163480624 +0000 UTC m=+2635.806641147"
Jul 07 03:02:03.170134 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:03.170087    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:02:07.169141 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:07.169108    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:02:07.171047 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:07.171020    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 07 03:02:09.064986 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:09.064931    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:02:17.170300 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:17.170258    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 07 03:02:19.064607 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:19.064560    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:02:27.170027 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:27.169951    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 07 03:02:29.064153 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:29.064106    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:02:37.170027 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:37.169983    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 07 03:02:39.064683 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:39.064627    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:02:47.170267 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:47.170216    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 07 03:02:49.064012 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:49.063968    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:02:57.169957 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:57.169907    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 07 03:02:59.064259 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:02:59.064119    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:03:07.170285 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:03:07.170228    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 07 03:03:08.041337 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:03:08.041299    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:03:08.046575 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:03:08.046545    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:03:09.074393 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:03:09.074360    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:03:09.082387 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:03:09.082359    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:03:17.169454 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:03:17.169405    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": dial tcp 10.134.0.53:8000: connect: connection refused"
Jul 07 03:03:27.180296 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:03:27.180263    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:03:27.189830 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:03:27.189801    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:04:03.571703 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:03.571663    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"]
Jul 07 03:04:13.550890 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:13.550856    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"]
Jul 07 03:04:18.595651 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:18.595593    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main" containerID="cri-o://9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce" gracePeriod=45
Jul 07 03:04:19.244648 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.244624    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:04:19.297557 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297467    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-dshm\") pod \"e71ac637-0424-4d41-b4ca-1577c77f6215\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") "
Jul 07 03:04:19.297557 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297543    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-model-cache\") pod \"e71ac637-0424-4d41-b4ca-1577c77f6215\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") "
Jul 07 03:04:19.297557 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297560    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e71ac637-0424-4d41-b4ca-1577c77f6215-cabundle-cert\") pod \"e71ac637-0424-4d41-b4ca-1577c77f6215\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") "
Jul 07 03:04:19.297844 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297582    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-kserve-provision-location\") pod \"e71ac637-0424-4d41-b4ca-1577c77f6215\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") "
Jul 07 03:04:19.297844 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297601    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-home\") pod \"e71ac637-0424-4d41-b4ca-1577c77f6215\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") "
Jul 07 03:04:19.297844 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297623    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71ac637-0424-4d41-b4ca-1577c77f6215-tls-certs\") pod \"e71ac637-0424-4d41-b4ca-1577c77f6215\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") "
Jul 07 03:04:19.297844 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297652    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-tmp-dir\") pod \"e71ac637-0424-4d41-b4ca-1577c77f6215\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") "
Jul 07 03:04:19.297844 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297701    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftjjh\" (UniqueName: \"kubernetes.io/projected/e71ac637-0424-4d41-b4ca-1577c77f6215-kube-api-access-ftjjh\") pod \"e71ac637-0424-4d41-b4ca-1577c77f6215\" (UID: \"e71ac637-0424-4d41-b4ca-1577c77f6215\") "
Jul 07 03:04:19.298099 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297883    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-model-cache" (OuterVolumeSpecName: "model-cache") pod "e71ac637-0424-4d41-b4ca-1577c77f6215" (UID: "e71ac637-0424-4d41-b4ca-1577c77f6215"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:19.298099 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297954    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-home" (OuterVolumeSpecName: "home") pod "e71ac637-0424-4d41-b4ca-1577c77f6215" (UID: "e71ac637-0424-4d41-b4ca-1577c77f6215"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:19.298099 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.297967    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e71ac637-0424-4d41-b4ca-1577c77f6215-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e71ac637-0424-4d41-b4ca-1577c77f6215" (UID: "e71ac637-0424-4d41-b4ca-1577c77f6215"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 03:04:19.299802 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.299777    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-dshm" (OuterVolumeSpecName: "dshm") pod "e71ac637-0424-4d41-b4ca-1577c77f6215" (UID: "e71ac637-0424-4d41-b4ca-1577c77f6215"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:19.300559 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.300532    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e71ac637-0424-4d41-b4ca-1577c77f6215-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e71ac637-0424-4d41-b4ca-1577c77f6215" (UID: "e71ac637-0424-4d41-b4ca-1577c77f6215"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:04:19.300668 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.300559    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e71ac637-0424-4d41-b4ca-1577c77f6215-kube-api-access-ftjjh" (OuterVolumeSpecName: "kube-api-access-ftjjh") pod "e71ac637-0424-4d41-b4ca-1577c77f6215" (UID: "e71ac637-0424-4d41-b4ca-1577c77f6215"). InnerVolumeSpecName "kube-api-access-ftjjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:04:19.308335 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.308305    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e71ac637-0424-4d41-b4ca-1577c77f6215" (UID: "e71ac637-0424-4d41-b4ca-1577c77f6215"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:19.366862 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.366821    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e71ac637-0424-4d41-b4ca-1577c77f6215" (UID: "e71ac637-0424-4d41-b4ca-1577c77f6215"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:19.398615 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.398581    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e71ac637-0424-4d41-b4ca-1577c77f6215-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:19.398760 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.398628    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:19.398760 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.398637    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftjjh\" (UniqueName: \"kubernetes.io/projected/e71ac637-0424-4d41-b4ca-1577c77f6215-kube-api-access-ftjjh\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:19.398760 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.398647    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:19.398760 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.398656    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:19.398760 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.398665    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e71ac637-0424-4d41-b4ca-1577c77f6215-cabundle-cert\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:19.398760 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.398673    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:19.398760 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.398681    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e71ac637-0424-4d41-b4ca-1577c77f6215-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:19.848057 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.848017    2543 generic.go:356] "Generic (PLEG): container finished" podID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerID="9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce" exitCode=0
Jul 07 03:04:19.848478 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.848078    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"
Jul 07 03:04:19.848478 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.848080    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" event={"ID":"e71ac637-0424-4d41-b4ca-1577c77f6215","Type":"ContainerDied","Data":"9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce"}
Jul 07 03:04:19.848478 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.848215    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" event={"ID":"e71ac637-0424-4d41-b4ca-1577c77f6215","Type":"ContainerDied","Data":"46c8c1c049538667352f51e4353a4b6a92290120adeee0496b818d8b3da3eb32"}
Jul 07 03:04:19.848478 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.848244    2543 scope.go:117] "RemoveContainer" containerID="9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce"
Jul 07 03:04:19.857421 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.857402    2543 scope.go:117] "RemoveContainer" containerID="a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445"
Jul 07 03:04:19.865611 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.865595    2543 scope.go:117] "RemoveContainer" containerID="9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce"
Jul 07 03:04:19.865865 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:04:19.865843    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce\": container with ID starting with 9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce not found: ID does not exist" containerID="9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce"
Jul 07 03:04:19.865913 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.865877    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce"} err="failed to get container status \"9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce\": rpc error: code = NotFound desc = could not find container \"9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce\": container with ID starting with 9007e2eefd42dd639f863d10a1387d6ad1bb4ef4bbc27451a1eeb3970817d2ce not found: ID does not exist"
Jul 07 03:04:19.865913 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.865895    2543 scope.go:117] "RemoveContainer" containerID="a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445"
Jul 07 03:04:19.866109 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:04:19.866093    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445\": container with ID starting with a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445 not found: ID does not exist" containerID="a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445"
Jul 07 03:04:19.866147 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.866114    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445"} err="failed to get container status \"a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445\": rpc error: code = NotFound desc = could not find container \"a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445\": container with ID starting with a4f7999b7c6a5b2476e93521e21f3a65d39dd6bfa9cadd637d8b13d1a21d3445 not found: ID does not exist"
Jul 07 03:04:19.873589 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.873565    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"]
Jul 07 03:04:19.883666 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.883633    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57"]
Jul 07 03:04:19.980430 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:19.980394    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" path="/var/lib/kubelet/pods/e71ac637-0424-4d41-b4ca-1577c77f6215/volumes"
Jul 07 03:04:20.182499 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:20.182445    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6f56f99c66-k5b57" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Jul 07 03:04:28.576021 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:28.575936    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" containerID="cri-o://2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402" gracePeriod=45
Jul 07 03:04:29.075143 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.075097    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": dial tcp 10.134.0.52:8000: connect: connection refused"
Jul 07 03:04:29.333370 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.333307    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:04:29.390462 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.390421    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-dshm\") pod \"20b2d517-4974-4a51-be75-9d4aaa456b00\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") "
Jul 07 03:04:29.390700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.390534    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-tmp-dir\") pod \"20b2d517-4974-4a51-be75-9d4aaa456b00\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") "
Jul 07 03:04:29.390700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.390560    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-model-cache\") pod \"20b2d517-4974-4a51-be75-9d4aaa456b00\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") "
Jul 07 03:04:29.390700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.390606    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-kserve-provision-location\") pod \"20b2d517-4974-4a51-be75-9d4aaa456b00\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") "
Jul 07 03:04:29.390700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.390634    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b2d517-4974-4a51-be75-9d4aaa456b00-tls-certs\") pod \"20b2d517-4974-4a51-be75-9d4aaa456b00\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") "
Jul 07 03:04:29.390700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.390674    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-home\") pod \"20b2d517-4974-4a51-be75-9d4aaa456b00\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") "
Jul 07 03:04:29.390966 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.390710    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj2lk\" (UniqueName: \"kubernetes.io/projected/20b2d517-4974-4a51-be75-9d4aaa456b00-kube-api-access-xj2lk\") pod \"20b2d517-4974-4a51-be75-9d4aaa456b00\" (UID: \"20b2d517-4974-4a51-be75-9d4aaa456b00\") "
Jul 07 03:04:29.390966 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.390830    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-model-cache" (OuterVolumeSpecName: "model-cache") pod "20b2d517-4974-4a51-be75-9d4aaa456b00" (UID: "20b2d517-4974-4a51-be75-9d4aaa456b00"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:29.391079 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.391009    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:29.391403 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.391345    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-home" (OuterVolumeSpecName: "home") pod "20b2d517-4974-4a51-be75-9d4aaa456b00" (UID: "20b2d517-4974-4a51-be75-9d4aaa456b00"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:29.393008 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.392977    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20b2d517-4974-4a51-be75-9d4aaa456b00-kube-api-access-xj2lk" (OuterVolumeSpecName: "kube-api-access-xj2lk") pod "20b2d517-4974-4a51-be75-9d4aaa456b00" (UID: "20b2d517-4974-4a51-be75-9d4aaa456b00"). InnerVolumeSpecName "kube-api-access-xj2lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:04:29.393109 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.393009    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20b2d517-4974-4a51-be75-9d4aaa456b00-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "20b2d517-4974-4a51-be75-9d4aaa456b00" (UID: "20b2d517-4974-4a51-be75-9d4aaa456b00"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:04:29.393109 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.393063    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-dshm" (OuterVolumeSpecName: "dshm") pod "20b2d517-4974-4a51-be75-9d4aaa456b00" (UID: "20b2d517-4974-4a51-be75-9d4aaa456b00"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:29.410807 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.410774    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "20b2d517-4974-4a51-be75-9d4aaa456b00" (UID: "20b2d517-4974-4a51-be75-9d4aaa456b00"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:29.465757 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.465715    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20b2d517-4974-4a51-be75-9d4aaa456b00" (UID: "20b2d517-4974-4a51-be75-9d4aaa456b00"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:04:29.491932 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.491907    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:29.491993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.491933    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/20b2d517-4974-4a51-be75-9d4aaa456b00-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:29.491993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.491944    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:29.491993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.491953    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2lk\" (UniqueName: \"kubernetes.io/projected/20b2d517-4974-4a51-be75-9d4aaa456b00-kube-api-access-xj2lk\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:29.491993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.491960    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:29.491993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.491971    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20b2d517-4974-4a51-be75-9d4aaa456b00-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:04:29.896095 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.896060    2543 generic.go:356] "Generic (PLEG): container finished" podID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerID="2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402" exitCode=0
Jul 07 03:04:29.896562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.896109    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" event={"ID":"20b2d517-4974-4a51-be75-9d4aaa456b00","Type":"ContainerDied","Data":"2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402"}
Jul 07 03:04:29.896562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.896160    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml" event={"ID":"20b2d517-4974-4a51-be75-9d4aaa456b00","Type":"ContainerDied","Data":"142aa501c2127339a63ebcc3a97d184006da428ec57c804f2490dd06940c12ca"}
Jul 07 03:04:29.896562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.896117    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"
Jul 07 03:04:29.896562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.896207    2543 scope.go:117] "RemoveContainer" containerID="2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402"
Jul 07 03:04:29.905133 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.905116    2543 scope.go:117] "RemoveContainer" containerID="1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9"
Jul 07 03:04:29.913596 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.913573    2543 scope.go:117] "RemoveContainer" containerID="2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402"
Jul 07 03:04:29.913855 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:04:29.913838    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402\": container with ID starting with 2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402 not found: ID does not exist" containerID="2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402"
Jul 07 03:04:29.913900 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.913864    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402"} err="failed to get container status \"2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402\": rpc error: code = NotFound desc = could not find container \"2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402\": container with ID starting with 2b5da885bac824c1d3f344e427ced69945c9a5d0d795b91332424e7cf2a4c402 not found: ID does not exist"
Jul 07 03:04:29.913900 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.913882    2543 scope.go:117] "RemoveContainer" containerID="1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9"
Jul 07 03:04:29.914100 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:04:29.914080    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9\": container with ID starting with 1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9 not found: ID does not exist" containerID="1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9"
Jul 07 03:04:29.914239 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.914110    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9"} err="failed to get container status \"1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9\": rpc error: code = NotFound desc = could not find container \"1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9\": container with ID starting with 1f7726ecb2ccdd8743ea7b175ee5c20d2a7f594d1d37b7cd5c0f097ffe70d5c9 not found: ID does not exist"
Jul 07 03:04:29.918315 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.918293    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"]
Jul 07 03:04:29.922362 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.922342    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-cd8b578dc-h8cml"]
Jul 07 03:04:29.979435 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:29.979397    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" path="/var/lib/kubelet/pods/20b2d517-4974-4a51-be75-9d4aaa456b00/volumes"
Jul 07 03:04:32.336859 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.336824    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"]
Jul 07 03:04:32.337266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.337125    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="storage-initializer"
Jul 07 03:04:32.337266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.337135    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="storage-initializer"
Jul 07 03:04:32.337266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.337144    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="storage-initializer"
Jul 07 03:04:32.337266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.337149    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="storage-initializer"
Jul 07 03:04:32.337266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.337163    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main"
Jul 07 03:04:32.337266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.337169    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main"
Jul 07 03:04:32.337266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.337178    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main"
Jul 07 03:04:32.337266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.337208    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main"
Jul 07 03:04:32.337266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.337268    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e71ac637-0424-4d41-b4ca-1577c77f6215" containerName="main"
Jul 07 03:04:32.337559 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.337276    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="20b2d517-4974-4a51-be75-9d4aaa456b00" containerName="main"
Jul 07 03:04:32.341782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.341764    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.344418 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.344391    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 07 03:04:32.344564 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.344419    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 07 03:04:32.344564 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.344463    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 07 03:04:32.344564 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.344519    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 03:04:32.344564 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.344529    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 07 03:04:32.344744 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.344716    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 03:04:32.352207 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.352162    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"]
Jul 07 03:04:32.419018 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.418977    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-home\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.419018 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.419018    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-dshm\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.419291 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.419049    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.419291 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.419136    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.419291 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.419164    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5159d30-9491-498f-aefb-2bb8a061c2f6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.419291 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.419238    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-model-cache\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.419291 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.419263    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5159d30-9491-498f-aefb-2bb8a061c2f6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.419291 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.419288    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7dpz\" (UniqueName: \"kubernetes.io/projected/e5159d30-9491-498f-aefb-2bb8a061c2f6-kube-api-access-n7dpz\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.520612 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.520564    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-home\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.520612 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.520618    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-dshm\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.520777 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.520642    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.520777 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.520682    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.520777 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.520700    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5159d30-9491-498f-aefb-2bb8a061c2f6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.520777 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.520727    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-model-cache\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.520777 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.520750    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5159d30-9491-498f-aefb-2bb8a061c2f6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.521014 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.520779    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7dpz\" (UniqueName: \"kubernetes.io/projected/e5159d30-9491-498f-aefb-2bb8a061c2f6-kube-api-access-n7dpz\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.521068 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.521044    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-home\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.521175 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.521143    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.521365 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.521271    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-model-cache\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.521365 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.521326    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.521455 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.521432    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5159d30-9491-498f-aefb-2bb8a061c2f6-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.522986 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.522968    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-dshm\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.523357 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.523335    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5159d30-9491-498f-aefb-2bb8a061c2f6-tls-certs\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.528733 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.528709    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7dpz\" (UniqueName: \"kubernetes.io/projected/e5159d30-9491-498f-aefb-2bb8a061c2f6-kube-api-access-n7dpz\") pod \"lora-multiple-adapters-test-kserve-764b5b74d7-v48lv\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.653548 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.653509    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:32.799845 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.799816    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"]
Jul 07 03:04:32.802312 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:04:32.802285    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5159d30_9491_498f_aefb_2bb8a061c2f6.slice/crio-1be95640f19a3bdddc6a0c61fa80a971422ebaae4176c6d904cb5a68dace11af WatchSource:0}: Error finding container 1be95640f19a3bdddc6a0c61fa80a971422ebaae4176c6d904cb5a68dace11af: Status 404 returned error can't find the container with id 1be95640f19a3bdddc6a0c61fa80a971422ebaae4176c6d904cb5a68dace11af
Jul 07 03:04:32.914158 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.913985    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" event={"ID":"e5159d30-9491-498f-aefb-2bb8a061c2f6","Type":"ContainerStarted","Data":"d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72"}
Jul 07 03:04:32.914158 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:32.914051    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" event={"ID":"e5159d30-9491-498f-aefb-2bb8a061c2f6","Type":"ContainerStarted","Data":"1be95640f19a3bdddc6a0c61fa80a971422ebaae4176c6d904cb5a68dace11af"}
Jul 07 03:04:37.940892 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:37.940861    2543 generic.go:356] "Generic (PLEG): container finished" podID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerID="d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72" exitCode=0
Jul 07 03:04:37.941328 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:37.940932    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" event={"ID":"e5159d30-9491-498f-aefb-2bb8a061c2f6","Type":"ContainerDied","Data":"d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72"}
Jul 07 03:04:38.947131 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:38.947094    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" event={"ID":"e5159d30-9491-498f-aefb-2bb8a061c2f6","Type":"ContainerStarted","Data":"9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7"}
Jul 07 03:04:38.971310 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:38.971181    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podStartSLOduration=2.548487823 podStartE2EDuration="6.971158641s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.422670818s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:04:38.968958883 +0000 UTC m=+2791.612119444" watchObservedRunningTime="2026-07-07 03:04:38.971158641 +0000 UTC m=+2791.614319164"
Jul 07 03:04:39.654514 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:39.654473    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:42.654283 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:42.654236    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:04:42.656041 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:42.656013    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 07 03:04:52.654871 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:04:52.654825    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 07 03:05:02.654298 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:02.654247    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 07 03:05:10.201825 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.201788    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"]
Jul 07 03:05:10.206110 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.206083    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.208663 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.208635    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 07 03:05:10.208877 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.208860    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-rhm9p\""
Jul 07 03:05:10.212788 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.212765    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"]
Jul 07 03:05:10.215965 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.215256    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.219974 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.219952    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"]
Jul 07 03:05:10.234778 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.234742    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"]
Jul 07 03:05:10.254768 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.254734    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.254961 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.254884    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.254961 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.254938    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.255093 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.254963    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.255093 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.254993    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6ngv\" (UniqueName: \"kubernetes.io/projected/abe18758-2bfd-4bc1-aca5-966142c9b83c-kube-api-access-l6ngv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.255093 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.255017    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.255093 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.255061    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe18758-2bfd-4bc1-aca5-966142c9b83c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.255336 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.255143    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.255336 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.255209    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bshxc\" (UniqueName: \"kubernetes.io/projected/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kube-api-access-bshxc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.255336 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.255251    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.255336 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.255279    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.255336 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.255327    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.255538 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.255352    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.255538 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.255380    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.356563 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356522    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bshxc\" (UniqueName: \"kubernetes.io/projected/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kube-api-access-bshxc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.356563 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356565    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.356828 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356584    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.356828 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356621    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.356828 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356641    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.356828 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356662    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.356828 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356692    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.356828 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356748    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.356828 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356812    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.357208 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356844    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.357208 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356884    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6ngv\" (UniqueName: \"kubernetes.io/projected/abe18758-2bfd-4bc1-aca5-966142c9b83c-kube-api-access-l6ngv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.357208 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356909    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.357208 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.356942    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe18758-2bfd-4bc1-aca5-966142c9b83c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.357208 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.357024    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.357208 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.357054    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.357208 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.357069    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.357208 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.357107    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.357648 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.357384    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.357648 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.357612    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.357857 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.357824    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.357949 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.357926    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.358216 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.358175    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.359709 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.359690    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe18758-2bfd-4bc1-aca5-966142c9b83c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.359810 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.359787    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.359910 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.359894    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.360055 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.360037    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.365418 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.365388    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6ngv\" (UniqueName: \"kubernetes.io/projected/abe18758-2bfd-4bc1-aca5-966142c9b83c-kube-api-access-l6ngv\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.365498 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.365461    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bshxc\" (UniqueName: \"kubernetes.io/projected/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kube-api-access-bshxc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.519018 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.518929    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:10.527741 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.527713    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:10.686634 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.686604    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"]
Jul 07 03:05:10.688846 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:05:10.688813    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabe18758_2bfd_4bc1_aca5_966142c9b83c.slice/crio-1714ca7b01f10e1afc4d51ed44b08ba7573f501b5a91ce9d36bc4cf70b46f368 WatchSource:0}: Error finding container 1714ca7b01f10e1afc4d51ed44b08ba7573f501b5a91ce9d36bc4cf70b46f368: Status 404 returned error can't find the container with id 1714ca7b01f10e1afc4d51ed44b08ba7573f501b5a91ce9d36bc4cf70b46f368
Jul 07 03:05:10.703014 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:10.702986    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"]
Jul 07 03:05:10.704253 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:05:10.704223    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe53d2e3_7b5c_48bd_9ba2_dde3fbcea2bf.slice/crio-db057e44e05d2db46507a0cbf0a49a67ea4a70bad84d986b5353625a51c6bfb5 WatchSource:0}: Error finding container db057e44e05d2db46507a0cbf0a49a67ea4a70bad84d986b5353625a51c6bfb5: Status 404 returned error can't find the container with id db057e44e05d2db46507a0cbf0a49a67ea4a70bad84d986b5353625a51c6bfb5
Jul 07 03:05:11.106204 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:11.106159    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" event={"ID":"abe18758-2bfd-4bc1-aca5-966142c9b83c","Type":"ContainerStarted","Data":"ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23"}
Jul 07 03:05:11.106530 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:11.106380    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:11.106530 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:11.106431    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" event={"ID":"abe18758-2bfd-4bc1-aca5-966142c9b83c","Type":"ContainerStarted","Data":"1714ca7b01f10e1afc4d51ed44b08ba7573f501b5a91ce9d36bc4cf70b46f368"}
Jul 07 03:05:11.108157 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:11.108129    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" event={"ID":"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf","Type":"ContainerStarted","Data":"2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21"}
Jul 07 03:05:11.108317 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:11.108180    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" event={"ID":"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf","Type":"ContainerStarted","Data":"db057e44e05d2db46507a0cbf0a49a67ea4a70bad84d986b5353625a51c6bfb5"}
Jul 07 03:05:12.119139 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:12.119106    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" event={"ID":"abe18758-2bfd-4bc1-aca5-966142c9b83c","Type":"ContainerStarted","Data":"39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963"}
Jul 07 03:05:12.655013 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:12.654952    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 07 03:05:15.143053 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:15.142959    2543 generic.go:356] "Generic (PLEG): container finished" podID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerID="39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963" exitCode=0
Jul 07 03:05:15.143053 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:15.143032    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" event={"ID":"abe18758-2bfd-4bc1-aca5-966142c9b83c","Type":"ContainerDied","Data":"39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963"}
Jul 07 03:05:16.151236 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:16.151167    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" event={"ID":"abe18758-2bfd-4bc1-aca5-966142c9b83c","Type":"ContainerStarted","Data":"881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9"}
Jul 07 03:05:16.174781 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:16.174721    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podStartSLOduration=2.465248485 podStartE2EDuration="6.17470395s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.709455465s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:05:16.171941825 +0000 UTC m=+2828.815102360" watchObservedRunningTime="2026-07-07 03:05:16.17470395 +0000 UTC m=+2828.817864470"
Jul 07 03:05:17.157472 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:17.157438    2543 generic.go:356] "Generic (PLEG): container finished" podID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerID="2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21" exitCode=0
Jul 07 03:05:17.157905 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:17.157513    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" event={"ID":"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf","Type":"ContainerDied","Data":"2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21"}
Jul 07 03:05:18.165065 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:18.165025    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" event={"ID":"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf","Type":"ContainerStarted","Data":"5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d"}
Jul 07 03:05:18.190461 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:18.190317    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podStartSLOduration=2.163267946 podStartE2EDuration="8.190264701s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.026996755s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:05:18.188883466 +0000 UTC m=+2830.832043988" watchObservedRunningTime="2026-07-07 03:05:18.190264701 +0000 UTC m=+2830.833425223"
Jul 07 03:05:20.519911 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:20.519862    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:20.520383 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:20.519942    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:20.521272 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:20.521240    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:05:20.528007 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:20.527977    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:20.528146 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:20.528041    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:05:20.529274 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:20.529241    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:05:20.538807 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:20.538787    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:05:22.654131 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:22.654081    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 07 03:05:30.519479 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:30.519425    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:05:30.528639 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:30.528600    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:05:32.654029 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:32.653979    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 07 03:05:40.520002 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:40.519951    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:05:40.528617 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:40.528569    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:05:42.654474 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:42.654421    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 07 03:05:50.520074 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:50.520024    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:05:50.528949 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:50.528910    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:05:52.654608 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:05:52.654554    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 07 03:06:00.519746 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:00.519635    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:06:00.528277 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:00.528237    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:06:02.664725 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:02.664686    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:06:02.672809 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:02.672781    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:06:10.520178 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:10.520130    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:06:10.528774 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:10.528727    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:06:20.519556 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:20.519500    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:06:20.528372 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:20.528328    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:06:30.520234 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:30.520148    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:06:30.528397 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:30.528327    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:06:40.519727 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:40.519668    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:06:40.528333 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:40.528302    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:06:50.520045 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:50.519998    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:06:50.528138 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:06:50.528093    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:07:00.519507 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:00.519457    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:07:00.528220 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:00.528177    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:07:03.470776 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:03.470733    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"]
Jul 07 03:07:10.520394 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:10.520343    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:07:10.528323 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:10.528289    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:07:18.494952 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:18.494882    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" containerID="cri-o://9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7" gracePeriod=45
Jul 07 03:07:18.665728 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:18.665687    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 07 03:07:19.666463 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.666421    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": dial tcp 10.134.0.54:8000: connect: connection refused"
Jul 07 03:07:19.666908 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.666581    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:07:19.767519 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.767493    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:07:19.830844 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.830810    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-dshm\") pod \"e5159d30-9491-498f-aefb-2bb8a061c2f6\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") "
Jul 07 03:07:19.830844 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.830844    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5159d30-9491-498f-aefb-2bb8a061c2f6-cabundle-cert\") pod \"e5159d30-9491-498f-aefb-2bb8a061c2f6\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") "
Jul 07 03:07:19.831137 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.830866    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-model-cache\") pod \"e5159d30-9491-498f-aefb-2bb8a061c2f6\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") "
Jul 07 03:07:19.831137 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.830891    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-home\") pod \"e5159d30-9491-498f-aefb-2bb8a061c2f6\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") "
Jul 07 03:07:19.831137 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.830917    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-kserve-provision-location\") pod \"e5159d30-9491-498f-aefb-2bb8a061c2f6\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") "
Jul 07 03:07:19.831137 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.831025    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-tmp-dir\") pod \"e5159d30-9491-498f-aefb-2bb8a061c2f6\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") "
Jul 07 03:07:19.831137 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.831057    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7dpz\" (UniqueName: \"kubernetes.io/projected/e5159d30-9491-498f-aefb-2bb8a061c2f6-kube-api-access-n7dpz\") pod \"e5159d30-9491-498f-aefb-2bb8a061c2f6\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") "
Jul 07 03:07:19.831137 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.831088    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5159d30-9491-498f-aefb-2bb8a061c2f6-tls-certs\") pod \"e5159d30-9491-498f-aefb-2bb8a061c2f6\" (UID: \"e5159d30-9491-498f-aefb-2bb8a061c2f6\") "
Jul 07 03:07:19.831491 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.831150    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5159d30-9491-498f-aefb-2bb8a061c2f6" (UID: "e5159d30-9491-498f-aefb-2bb8a061c2f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:07:19.831491 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.831279    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-home" (OuterVolumeSpecName: "home") pod "e5159d30-9491-498f-aefb-2bb8a061c2f6" (UID: "e5159d30-9491-498f-aefb-2bb8a061c2f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:07:19.831491 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.831290    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5159d30-9491-498f-aefb-2bb8a061c2f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e5159d30-9491-498f-aefb-2bb8a061c2f6" (UID: "e5159d30-9491-498f-aefb-2bb8a061c2f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 07 03:07:19.831491 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.831398    2543 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e5159d30-9491-498f-aefb-2bb8a061c2f6-cabundle-cert\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:07:19.831491 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.831416    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:07:19.831491 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.831429    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:07:19.834067 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.834000    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5159d30-9491-498f-aefb-2bb8a061c2f6-kube-api-access-n7dpz" (OuterVolumeSpecName: "kube-api-access-n7dpz") pod "e5159d30-9491-498f-aefb-2bb8a061c2f6" (UID: "e5159d30-9491-498f-aefb-2bb8a061c2f6"). InnerVolumeSpecName "kube-api-access-n7dpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:07:19.834219 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.834106    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5159d30-9491-498f-aefb-2bb8a061c2f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5159d30-9491-498f-aefb-2bb8a061c2f6" (UID: "e5159d30-9491-498f-aefb-2bb8a061c2f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:07:19.834219 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.834115    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-dshm" (OuterVolumeSpecName: "dshm") pod "e5159d30-9491-498f-aefb-2bb8a061c2f6" (UID: "e5159d30-9491-498f-aefb-2bb8a061c2f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:07:19.847353 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.847306    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5159d30-9491-498f-aefb-2bb8a061c2f6" (UID: "e5159d30-9491-498f-aefb-2bb8a061c2f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:07:19.851834 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.851808    2543 generic.go:356] "Generic (PLEG): container finished" podID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerID="9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7" exitCode=0
Jul 07 03:07:19.851932 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.851849    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" event={"ID":"e5159d30-9491-498f-aefb-2bb8a061c2f6","Type":"ContainerDied","Data":"9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7"}
Jul 07 03:07:19.851932 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.851880    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"
Jul 07 03:07:19.851932 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.851898    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv" event={"ID":"e5159d30-9491-498f-aefb-2bb8a061c2f6","Type":"ContainerDied","Data":"1be95640f19a3bdddc6a0c61fa80a971422ebaae4176c6d904cb5a68dace11af"}
Jul 07 03:07:19.851932 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.851923    2543 scope.go:117] "RemoveContainer" containerID="9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7"
Jul 07 03:07:19.862402 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.862375    2543 scope.go:117] "RemoveContainer" containerID="d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72"
Jul 07 03:07:19.870983 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.870964    2543 scope.go:117] "RemoveContainer" containerID="9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7"
Jul 07 03:07:19.871338 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:07:19.871314    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7\": container with ID starting with 9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7 not found: ID does not exist" containerID="9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7"
Jul 07 03:07:19.871436 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.871346    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7"} err="failed to get container status \"9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7\": rpc error: code = NotFound desc = could not find container \"9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7\": container with ID starting with 9a738ec7ef3e5348bad1df650972c7ccf5a69c0175740e092d93f89f7f391ee7 not found: ID does not exist"
Jul 07 03:07:19.871436 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.871376    2543 scope.go:117] "RemoveContainer" containerID="d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72"
Jul 07 03:07:19.871647 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:07:19.871626    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72\": container with ID starting with d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72 not found: ID does not exist" containerID="d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72"
Jul 07 03:07:19.871713 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.871658    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72"} err="failed to get container status \"d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72\": rpc error: code = NotFound desc = could not find container \"d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72\": container with ID starting with d94a20115a85dfb3a55dfe1dde278acbb08522fa4e635b64dc80578d54af5e72 not found: ID does not exist"
Jul 07 03:07:19.903237 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.903173    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e5159d30-9491-498f-aefb-2bb8a061c2f6" (UID: "e5159d30-9491-498f-aefb-2bb8a061c2f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:07:19.932329 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.932289    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:07:19.932329 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.932328    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7dpz\" (UniqueName: \"kubernetes.io/projected/e5159d30-9491-498f-aefb-2bb8a061c2f6-kube-api-access-n7dpz\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:07:19.932488 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.932338    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5159d30-9491-498f-aefb-2bb8a061c2f6-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:07:19.932488 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.932348    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:07:19.932488 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:19.932356    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e5159d30-9491-498f-aefb-2bb8a061c2f6-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:07:20.168952 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:20.168920    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"]
Jul 07 03:07:20.172201 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:20.172165    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-764b5b74d7-v48lv"]
Jul 07 03:07:20.519622 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:20.519509    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:07:20.528268 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:20.528234    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:07:21.981288 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:21.981255    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" path="/var/lib/kubelet/pods/e5159d30-9491-498f-aefb-2bb8a061c2f6/volumes"
Jul 07 03:07:27.982758 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:27.982682    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"]
Jul 07 03:07:27.983201 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:27.983006    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="storage-initializer"
Jul 07 03:07:27.983201 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:27.983016    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="storage-initializer"
Jul 07 03:07:27.983201 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:27.983023    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main"
Jul 07 03:07:27.983201 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:27.983029    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main"
Jul 07 03:07:27.983201 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:27.983091    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5159d30-9491-498f-aefb-2bb8a061c2f6" containerName="main"
Jul 07 03:07:27.986033 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:27.986016    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:27.988411 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:27.988390    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 07 03:07:27.996875 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:27.996849    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"]
Jul 07 03:07:28.109315 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.109282    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-dshm\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.109548 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.109341    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-model-cache\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.109548 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.109357    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9g4n\" (UniqueName: \"kubernetes.io/projected/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kube-api-access-z9g4n\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.109548 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.109454    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-home\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.109548 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.109504    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kserve-provision-location\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.109548 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.109533    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tmp-dir\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.109548 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.109553    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tls-certs\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211014 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.210968    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tmp-dir\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211014 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.211004    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tls-certs\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211331 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.211055    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-dshm\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211331 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.211085    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-model-cache\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211331 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.211108    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9g4n\" (UniqueName: \"kubernetes.io/projected/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kube-api-access-z9g4n\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211331 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.211146    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-home\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211331 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.211213    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kserve-provision-location\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211603 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.211458    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tmp-dir\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211671 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.211601    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-model-cache\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211671 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.211643    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-home\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.211960 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.211854    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kserve-provision-location\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.213813 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.213787    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-dshm\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.214124 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.214103    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tls-certs\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.219531 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.219504    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9g4n\" (UniqueName: \"kubernetes.io/projected/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kube-api-access-z9g4n\") pod \"tls-verification-test-kserve-66dc875567-8hbzr\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.296993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.296898    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:28.448535 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.448508    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"]
Jul 07 03:07:28.450938 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:07:28.450913    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576d14b8_3288_4dac_bbc8_d9ff8e2e9563.slice/crio-970f2b599c76279ebbe6fafa0f49c3a16228eae76148366c6b347853a017cc83 WatchSource:0}: Error finding container 970f2b599c76279ebbe6fafa0f49c3a16228eae76148366c6b347853a017cc83: Status 404 returned error can't find the container with id 970f2b599c76279ebbe6fafa0f49c3a16228eae76148366c6b347853a017cc83
Jul 07 03:07:28.904068 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.904018    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" event={"ID":"576d14b8-3288-4dac-bbc8-d9ff8e2e9563","Type":"ContainerStarted","Data":"335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e"}
Jul 07 03:07:28.904286 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:28.904095    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" event={"ID":"576d14b8-3288-4dac-bbc8-d9ff8e2e9563","Type":"ContainerStarted","Data":"970f2b599c76279ebbe6fafa0f49c3a16228eae76148366c6b347853a017cc83"}
Jul 07 03:07:30.519687 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:30.519633    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:07:30.528425 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:30.528381    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:07:32.925210 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:32.925156    2543 generic.go:356] "Generic (PLEG): container finished" podID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerID="335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e" exitCode=0
Jul 07 03:07:32.925636 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:32.925225    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" event={"ID":"576d14b8-3288-4dac-bbc8-d9ff8e2e9563","Type":"ContainerDied","Data":"335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e"}
Jul 07 03:07:33.932540 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:33.932492    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" event={"ID":"576d14b8-3288-4dac-bbc8-d9ff8e2e9563","Type":"ContainerStarted","Data":"67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf"}
Jul 07 03:07:33.957991 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:33.957904    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podStartSLOduration=2.743568134 podStartE2EDuration="6.957851571s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.214283437s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:07:33.956262307 +0000 UTC m=+2966.599422831" watchObservedRunningTime="2026-07-07 03:07:33.957851571 +0000 UTC m=+2966.601012097"
Jul 07 03:07:34.297381 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:34.297290    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:38.298105 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:38.298053    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:07:38.299953 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:38.299923    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:07:40.519407 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:40.519357    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:07:40.528683 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:40.528632    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:07:48.297707 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:48.297654    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:07:50.520237 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:50.520178    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:07:50.528710 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:50.528677    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:07:58.298026 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:07:58.297968    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:08:00.520301 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:00.520256    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8001/health\": dial tcp 10.134.0.55:8001: connect: connection refused"
Jul 07 03:08:00.528706 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:00.528672    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": dial tcp 10.134.0.56:8000: connect: connection refused"
Jul 07 03:08:08.071259 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:08.071226    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:08:08.085021 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:08.084991    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:08:08.297527 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:08.297483    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:08:10.531857 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:10.531827    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:08:10.540108 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:10.540066    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:08:10.544853 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:10.544831    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:08:10.552491 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:10.552475    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:08:18.297703 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:18.297645    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:08:20.771642 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:20.771603    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"]
Jul 07 03:08:20.776674 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:20.776647    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"]
Jul 07 03:08:28.297591 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:28.297548    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:08:35.799121 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:35.799074    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main" containerID="cri-o://5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d" gracePeriod=45
Jul 07 03:08:35.802103 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:35.802063    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main" containerID="cri-o://881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9" gracePeriod=45
Jul 07 03:08:36.624618 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.624549    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="llm-d-routing-sidecar" containerID="cri-o://ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23" gracePeriod=45
Jul 07 03:08:36.775614 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.775589    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:08:36.845315 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.845238    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tls-certs\") pod \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") "
Jul 07 03:08:36.845737 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.845316    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kserve-provision-location\") pod \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") "
Jul 07 03:08:36.845737 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.845348    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tmp-dir\") pod \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") "
Jul 07 03:08:36.845737 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.845368    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-home\") pod \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") "
Jul 07 03:08:36.845737 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.845385    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-dshm\") pod \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") "
Jul 07 03:08:36.845737 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.845442    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-model-cache\") pod \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") "
Jul 07 03:08:36.845737 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.845470    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bshxc\" (UniqueName: \"kubernetes.io/projected/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kube-api-access-bshxc\") pod \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\" (UID: \"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf\") "
Jul 07 03:08:36.846056 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.845877    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" (UID: "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:36.846131 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.846088    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-home" (OuterVolumeSpecName: "home") pod "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" (UID: "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:36.847873 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.847846    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-dshm" (OuterVolumeSpecName: "dshm") pod "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" (UID: "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:36.848059 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.848039    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" (UID: "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:08:36.848228 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.848208    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kube-api-access-bshxc" (OuterVolumeSpecName: "kube-api-access-bshxc") pod "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" (UID: "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf"). InnerVolumeSpecName "kube-api-access-bshxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:08:36.859894 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.859840    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" (UID: "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:36.909462 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.909432    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" (UID: "be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:36.911087 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.911068    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:08:36.945994 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.945962    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6ngv\" (UniqueName: \"kubernetes.io/projected/abe18758-2bfd-4bc1-aca5-966142c9b83c-kube-api-access-l6ngv\") pod \"abe18758-2bfd-4bc1-aca5-966142c9b83c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") "
Jul 07 03:08:36.946204 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946013    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-model-cache\") pod \"abe18758-2bfd-4bc1-aca5-966142c9b83c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") "
Jul 07 03:08:36.946204 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946034    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-dshm\") pod \"abe18758-2bfd-4bc1-aca5-966142c9b83c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") "
Jul 07 03:08:36.946392 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946359    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-model-cache" (OuterVolumeSpecName: "model-cache") pod "abe18758-2bfd-4bc1-aca5-966142c9b83c" (UID: "abe18758-2bfd-4bc1-aca5-966142c9b83c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:36.946392 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946370    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-tmp-dir\") pod \"abe18758-2bfd-4bc1-aca5-966142c9b83c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") "
Jul 07 03:08:36.946589 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946453    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-kserve-provision-location\") pod \"abe18758-2bfd-4bc1-aca5-966142c9b83c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") "
Jul 07 03:08:36.946589 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946552    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe18758-2bfd-4bc1-aca5-966142c9b83c-tls-certs\") pod \"abe18758-2bfd-4bc1-aca5-966142c9b83c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") "
Jul 07 03:08:36.946589 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946586    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-home\") pod \"abe18758-2bfd-4bc1-aca5-966142c9b83c\" (UID: \"abe18758-2bfd-4bc1-aca5-966142c9b83c\") "
Jul 07 03:08:36.946891 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946862    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:36.947654 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946899    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:36.947654 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946914    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:36.947654 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946928    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:36.947654 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946943    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:36.947654 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946955    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:36.947654 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946967    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:36.947654 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.946981    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bshxc\" (UniqueName: \"kubernetes.io/projected/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf-kube-api-access-bshxc\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:36.947654 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.947538    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-home" (OuterVolumeSpecName: "home") pod "abe18758-2bfd-4bc1-aca5-966142c9b83c" (UID: "abe18758-2bfd-4bc1-aca5-966142c9b83c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:36.948778 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.948750    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-dshm" (OuterVolumeSpecName: "dshm") pod "abe18758-2bfd-4bc1-aca5-966142c9b83c" (UID: "abe18758-2bfd-4bc1-aca5-966142c9b83c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:36.948874 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.948789    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe18758-2bfd-4bc1-aca5-966142c9b83c-kube-api-access-l6ngv" (OuterVolumeSpecName: "kube-api-access-l6ngv") pod "abe18758-2bfd-4bc1-aca5-966142c9b83c" (UID: "abe18758-2bfd-4bc1-aca5-966142c9b83c"). InnerVolumeSpecName "kube-api-access-l6ngv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:08:36.950585 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.950554    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe18758-2bfd-4bc1-aca5-966142c9b83c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "abe18758-2bfd-4bc1-aca5-966142c9b83c" (UID: "abe18758-2bfd-4bc1-aca5-966142c9b83c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:08:36.959917 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.959891    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "abe18758-2bfd-4bc1-aca5-966142c9b83c" (UID: "abe18758-2bfd-4bc1-aca5-966142c9b83c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:36.986652 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:36.986615    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "abe18758-2bfd-4bc1-aca5-966142c9b83c" (UID: "abe18758-2bfd-4bc1-aca5-966142c9b83c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:08:37.048104 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.048066    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:37.048104 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.048098    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:37.048104 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.048112    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:37.048347 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.048122    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abe18758-2bfd-4bc1-aca5-966142c9b83c-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:37.048347 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.048130    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abe18758-2bfd-4bc1-aca5-966142c9b83c-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:37.048347 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.048138    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6ngv\" (UniqueName: \"kubernetes.io/projected/abe18758-2bfd-4bc1-aca5-966142c9b83c-kube-api-access-l6ngv\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:08:37.287723 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.287685    2543 generic.go:356] "Generic (PLEG): container finished" podID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerID="5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d" exitCode=0
Jul 07 03:08:37.287929 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.287751    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"
Jul 07 03:08:37.287929 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.287743    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" event={"ID":"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf","Type":"ContainerDied","Data":"5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d"}
Jul 07 03:08:37.287929 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.287902    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq" event={"ID":"be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf","Type":"ContainerDied","Data":"db057e44e05d2db46507a0cbf0a49a67ea4a70bad84d986b5353625a51c6bfb5"}
Jul 07 03:08:37.288132 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.287940    2543 scope.go:117] "RemoveContainer" containerID="5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d"
Jul 07 03:08:37.289913 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.289888    2543 generic.go:356] "Generic (PLEG): container finished" podID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerID="881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9" exitCode=0
Jul 07 03:08:37.289913 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.289913    2543 generic.go:356] "Generic (PLEG): container finished" podID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerID="ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23" exitCode=0
Jul 07 03:08:37.290099 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.289932    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" event={"ID":"abe18758-2bfd-4bc1-aca5-966142c9b83c","Type":"ContainerDied","Data":"881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9"}
Jul 07 03:08:37.290099 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.289959    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"
Jul 07 03:08:37.290099 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.290002    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" event={"ID":"abe18758-2bfd-4bc1-aca5-966142c9b83c","Type":"ContainerDied","Data":"ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23"}
Jul 07 03:08:37.290099 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.290032    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c" event={"ID":"abe18758-2bfd-4bc1-aca5-966142c9b83c","Type":"ContainerDied","Data":"1714ca7b01f10e1afc4d51ed44b08ba7573f501b5a91ce9d36bc4cf70b46f368"}
Jul 07 03:08:37.298691 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.298676    2543 scope.go:117] "RemoveContainer" containerID="2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21"
Jul 07 03:08:37.307454 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.307435    2543 scope.go:117] "RemoveContainer" containerID="5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d"
Jul 07 03:08:37.307724 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:08:37.307707    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d\": container with ID starting with 5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d not found: ID does not exist" containerID="5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d"
Jul 07 03:08:37.307772 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.307733    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d"} err="failed to get container status \"5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d\": rpc error: code = NotFound desc = could not find container \"5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d\": container with ID starting with 5440cbaaebdecfbcb0b67e52731848508af0e1b1da29336e48b2af7a046ae00d not found: ID does not exist"
Jul 07 03:08:37.307772 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.307749    2543 scope.go:117] "RemoveContainer" containerID="2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21"
Jul 07 03:08:37.307997 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:08:37.307982    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21\": container with ID starting with 2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21 not found: ID does not exist" containerID="2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21"
Jul 07 03:08:37.308061 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.308001    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21"} err="failed to get container status \"2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21\": rpc error: code = NotFound desc = could not find container \"2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21\": container with ID starting with 2b69b2d9d8e7dbe51741a75db936631a46e7ad77ccb4f69c46e9674b15726a21 not found: ID does not exist"
Jul 07 03:08:37.308061 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.308013    2543 scope.go:117] "RemoveContainer" containerID="881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9"
Jul 07 03:08:37.310845 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.310822    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"]
Jul 07 03:08:37.316354 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.316332    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-6dgs5wq"]
Jul 07 03:08:37.317331 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.317315    2543 scope.go:117] "RemoveContainer" containerID="39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963"
Jul 07 03:08:37.325294 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.325275    2543 scope.go:117] "RemoveContainer" containerID="ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23"
Jul 07 03:08:37.327072 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.327053    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"]
Jul 07 03:08:37.333710 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.333688    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-f99ccccc7-wlb6c"]
Jul 07 03:08:37.334262 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.334245    2543 scope.go:117] "RemoveContainer" containerID="881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9"
Jul 07 03:08:37.334529 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:08:37.334511    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9\": container with ID starting with 881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9 not found: ID does not exist" containerID="881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9"
Jul 07 03:08:37.334609 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.334537    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9"} err="failed to get container status \"881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9\": rpc error: code = NotFound desc = could not find container \"881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9\": container with ID starting with 881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9 not found: ID does not exist"
Jul 07 03:08:37.334609 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.334556    2543 scope.go:117] "RemoveContainer" containerID="39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963"
Jul 07 03:08:37.334844 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:08:37.334827    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963\": container with ID starting with 39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963 not found: ID does not exist" containerID="39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963"
Jul 07 03:08:37.334887 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.334849    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963"} err="failed to get container status \"39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963\": rpc error: code = NotFound desc = could not find container \"39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963\": container with ID starting with 39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963 not found: ID does not exist"
Jul 07 03:08:37.334887 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.334865    2543 scope.go:117] "RemoveContainer" containerID="ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23"
Jul 07 03:08:37.335109 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:08:37.335093    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23\": container with ID starting with ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23 not found: ID does not exist" containerID="ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23"
Jul 07 03:08:37.335157 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.335115    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23"} err="failed to get container status \"ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23\": rpc error: code = NotFound desc = could not find container \"ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23\": container with ID starting with ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23 not found: ID does not exist"
Jul 07 03:08:37.335157 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.335130    2543 scope.go:117] "RemoveContainer" containerID="881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9"
Jul 07 03:08:37.335448 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.335433    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9"} err="failed to get container status \"881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9\": rpc error: code = NotFound desc = could not find container \"881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9\": container with ID starting with 881e5175932093c461a2dfc82ecc7f990c622b9acc16cff2cf8855d3d234b3a9 not found: ID does not exist"
Jul 07 03:08:37.335501 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.335448    2543 scope.go:117] "RemoveContainer" containerID="39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963"
Jul 07 03:08:37.335681 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.335664    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963"} err="failed to get container status \"39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963\": rpc error: code = NotFound desc = could not find container \"39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963\": container with ID starting with 39054f6d6cb08e77107ffb608527bdb41d02ea5a5efa600d3edce897b7983963 not found: ID does not exist"
Jul 07 03:08:37.335728 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.335680    2543 scope.go:117] "RemoveContainer" containerID="ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23"
Jul 07 03:08:37.335921 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.335902    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23"} err="failed to get container status \"ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23\": rpc error: code = NotFound desc = could not find container \"ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23\": container with ID starting with ee6d1b4e45698267d3fec74ba013b9c11cf084cf4830b2d55fbaf1f354d8fa23 not found: ID does not exist"
Jul 07 03:08:37.981244 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.981212    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" path="/var/lib/kubelet/pods/abe18758-2bfd-4bc1-aca5-966142c9b83c/volumes"
Jul 07 03:08:37.981695 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:37.981681    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" path="/var/lib/kubelet/pods/be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf/volumes"
Jul 07 03:08:38.297375 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:38.297291    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:08:47.478852 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.478810    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"]
Jul 07 03:08:47.479350 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479298    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="llm-d-routing-sidecar"
Jul 07 03:08:47.479350 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479318    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="llm-d-routing-sidecar"
Jul 07 03:08:47.479350 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479336    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main"
Jul 07 03:08:47.479350 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479344    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main"
Jul 07 03:08:47.479562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479359    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="storage-initializer"
Jul 07 03:08:47.479562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479367    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="storage-initializer"
Jul 07 03:08:47.479562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479376    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main"
Jul 07 03:08:47.479562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479385    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main"
Jul 07 03:08:47.479562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479409    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="storage-initializer"
Jul 07 03:08:47.479562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479419    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="storage-initializer"
Jul 07 03:08:47.479562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479513    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="main"
Jul 07 03:08:47.479562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479524    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="be53d2e3-7b5c-48bd-9ba2-dde3fbcea2bf" containerName="main"
Jul 07 03:08:47.479562 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.479538    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe18758-2bfd-4bc1-aca5-966142c9b83c" containerName="llm-d-routing-sidecar"
Jul 07 03:08:47.482942 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.482914    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.486247 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.486221    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 07 03:08:47.486351 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.486317    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-rbqq4\""
Jul 07 03:08:47.497984 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.497963    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"]
Jul 07 03:08:47.501133 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.501118    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.505006 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.504982    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"]
Jul 07 03:08:47.521751 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.521726    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"]
Jul 07 03:08:47.540332 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540298    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-dshm\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.540517 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540362    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.540517 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540412    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.540517 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540485    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.540692 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540543    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.540692 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540569    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.540692 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540592    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xjx8\" (UniqueName: \"kubernetes.io/projected/beeace29-430c-4b69-8a70-9bdcd069b691-kube-api-access-7xjx8\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.540692 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540628    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8gf4\" (UniqueName: \"kubernetes.io/projected/7d82ee77-9347-440d-a18b-f958769e3180-kube-api-access-d8gf4\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.540692 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540673    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d82ee77-9347-440d-a18b-f958769e3180-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.540975 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540696    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.540975 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540718    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.540975 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540744    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.540975 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540779    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-home\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.540975 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.540835    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beeace29-430c-4b69-8a70-9bdcd069b691-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.641345 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641306    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d82ee77-9347-440d-a18b-f958769e3180-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.641345 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641345    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.641608 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641362    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.641608 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641392    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.641608 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641522    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-home\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.641608 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641581    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beeace29-430c-4b69-8a70-9bdcd069b691-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.641816 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641616    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-dshm\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.641816 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641646    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.641816 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641692    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.641816 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641784    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.641816 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641783    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-model-cache\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.642067 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.641844    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-home\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.642067 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.642029    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.642067 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.642053    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.642416 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.642131    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.642416 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.642173    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.642416 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.642216    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.642416 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.642229    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.642416 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.642344    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.642685 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.642379    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xjx8\" (UniqueName: \"kubernetes.io/projected/beeace29-430c-4b69-8a70-9bdcd069b691-kube-api-access-7xjx8\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.642685 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.642516    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.642685 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.642518    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8gf4\" (UniqueName: \"kubernetes.io/projected/7d82ee77-9347-440d-a18b-f958769e3180-kube-api-access-d8gf4\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.644127 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.644098    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-dshm\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.644382 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.644366    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d82ee77-9347-440d-a18b-f958769e3180-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.644523 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.644502    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beeace29-430c-4b69-8a70-9bdcd069b691-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.644917 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.644903    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.651556 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.651533    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8gf4\" (UniqueName: \"kubernetes.io/projected/7d82ee77-9347-440d-a18b-f958769e3180-kube-api-access-d8gf4\") pod \"custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.651687 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.651572    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xjx8\" (UniqueName: \"kubernetes.io/projected/beeace29-430c-4b69-8a70-9bdcd069b691-kube-api-access-7xjx8\") pod \"custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.792933 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.792835    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:47.813231 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.813173    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:47.955814 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.955782    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"]
Jul 07 03:08:47.959307 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:08:47.959269    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d82ee77_9347_440d_a18b_f958769e3180.slice/crio-6fe9c98b1ccdac0fc73f41f0fe4c353588be01d5b088219736f23242a70c2368 WatchSource:0}: Error finding container 6fe9c98b1ccdac0fc73f41f0fe4c353588be01d5b088219736f23242a70c2368: Status 404 returned error can't find the container with id 6fe9c98b1ccdac0fc73f41f0fe4c353588be01d5b088219736f23242a70c2368
Jul 07 03:08:47.981089 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:47.981063    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"]
Jul 07 03:08:47.981827 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:08:47.981800    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeeace29_430c_4b69_8a70_9bdcd069b691.slice/crio-9e6a59aa0c8723c4ca55c870cc1de5651f99377518356985d4d33491bbe0af2a WatchSource:0}: Error finding container 9e6a59aa0c8723c4ca55c870cc1de5651f99377518356985d4d33491bbe0af2a: Status 404 returned error can't find the container with id 9e6a59aa0c8723c4ca55c870cc1de5651f99377518356985d4d33491bbe0af2a
Jul 07 03:08:48.297511 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:48.297459    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:08:48.346990 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:48.346946    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" event={"ID":"7d82ee77-9347-440d-a18b-f958769e3180","Type":"ContainerStarted","Data":"4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7"}
Jul 07 03:08:48.347177 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:48.347060    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" event={"ID":"7d82ee77-9347-440d-a18b-f958769e3180","Type":"ContainerStarted","Data":"6fe9c98b1ccdac0fc73f41f0fe4c353588be01d5b088219736f23242a70c2368"}
Jul 07 03:08:48.347177 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:48.347101    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:48.349149 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:48.349123    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" event={"ID":"beeace29-430c-4b69-8a70-9bdcd069b691","Type":"ContainerStarted","Data":"4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b"}
Jul 07 03:08:48.349282 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:48.349167    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" event={"ID":"beeace29-430c-4b69-8a70-9bdcd069b691","Type":"ContainerStarted","Data":"9e6a59aa0c8723c4ca55c870cc1de5651f99377518356985d4d33491bbe0af2a"}
Jul 07 03:08:49.361724 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:49.361686    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" event={"ID":"7d82ee77-9347-440d-a18b-f958769e3180","Type":"ContainerStarted","Data":"56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b"}
Jul 07 03:08:52.382234 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:52.382179    2543 generic.go:356] "Generic (PLEG): container finished" podID="7d82ee77-9347-440d-a18b-f958769e3180" containerID="56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b" exitCode=0
Jul 07 03:08:52.382707 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:52.382264    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" event={"ID":"7d82ee77-9347-440d-a18b-f958769e3180","Type":"ContainerDied","Data":"56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b"}
Jul 07 03:08:52.383852 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:52.383831    2543 generic.go:356] "Generic (PLEG): container finished" podID="beeace29-430c-4b69-8a70-9bdcd069b691" containerID="4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b" exitCode=0
Jul 07 03:08:52.383942 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:52.383865    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" event={"ID":"beeace29-430c-4b69-8a70-9bdcd069b691","Type":"ContainerDied","Data":"4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b"}
Jul 07 03:08:53.392037 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:53.391988    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" event={"ID":"7d82ee77-9347-440d-a18b-f958769e3180","Type":"ContainerStarted","Data":"ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc"}
Jul 07 03:08:53.394394 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:53.394361    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" event={"ID":"beeace29-430c-4b69-8a70-9bdcd069b691","Type":"ContainerStarted","Data":"de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158"}
Jul 07 03:08:53.418102 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:53.418023    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podStartSLOduration=2.755232951 podStartE2EDuration="6.418000619s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.662767668s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:08:53.416226165 +0000 UTC m=+3046.059386688" watchObservedRunningTime="2026-07-07 03:08:53.418000619 +0000 UTC m=+3046.061161143"
Jul 07 03:08:53.436250 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:53.436154    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podStartSLOduration=2.952936739 podStartE2EDuration="6.436133348s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.483196609s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:08:53.435020018 +0000 UTC m=+3046.078180541" watchObservedRunningTime="2026-07-07 03:08:53.436133348 +0000 UTC m=+3046.079293870"
Jul 07 03:08:57.793494 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:57.793394    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:57.793494 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:57.793475    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:08:57.794756 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:57.794721    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:08:57.814031 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:57.813989    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:57.814236 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:57.814073    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:08:57.815571 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:57.815537    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:08:58.297563 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:08:58.297515    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:09:07.793925 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:07.793862    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:09:07.812344 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:07.812314    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:09:07.813661 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:07.813631    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:09:08.297416 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:08.297357    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:09:17.793923 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:17.793874    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:09:17.814005 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:17.813965    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:09:18.297661 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:18.297607    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:09:27.793963 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:27.793906    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:09:27.813736 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:27.813701    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:09:28.298252 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:28.298207    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": dial tcp 10.134.0.57:8000: connect: connection refused"
Jul 07 03:09:37.793768 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:37.793722    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:09:37.813577 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:37.813534    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:09:38.308936 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:38.308908    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:09:38.321505 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:38.321461    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:09:43.571644 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:43.571608    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"]
Jul 07 03:09:47.794039 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:47.793994    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:09:47.813614 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:47.813567    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:09:57.793348 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:57.793286    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:09:57.814639 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:57.814593    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:09:58.608853 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:58.608799    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" containerID="cri-o://67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf" gracePeriod=45
Jul 07 03:09:59.382894 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.382856    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:09:59.544481 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.544383    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tmp-dir\") pod \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") "
Jul 07 03:09:59.544481 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.544438    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-home\") pod \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") "
Jul 07 03:09:59.544481 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.544467    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tls-certs\") pod \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") "
Jul 07 03:09:59.544782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.544532    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-model-cache\") pod \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") "
Jul 07 03:09:59.544782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.544599    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kserve-provision-location\") pod \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") "
Jul 07 03:09:59.544782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.544633    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-dshm\") pod \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") "
Jul 07 03:09:59.544782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.544691    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9g4n\" (UniqueName: \"kubernetes.io/projected/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kube-api-access-z9g4n\") pod \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\" (UID: \"576d14b8-3288-4dac-bbc8-d9ff8e2e9563\") "
Jul 07 03:09:59.544782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.544714    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-model-cache" (OuterVolumeSpecName: "model-cache") pod "576d14b8-3288-4dac-bbc8-d9ff8e2e9563" (UID: "576d14b8-3288-4dac-bbc8-d9ff8e2e9563"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:09:59.545036 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.544936    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:09:59.545169 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.545145    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-home" (OuterVolumeSpecName: "home") pod "576d14b8-3288-4dac-bbc8-d9ff8e2e9563" (UID: "576d14b8-3288-4dac-bbc8-d9ff8e2e9563"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:09:59.547420 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.547368    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-dshm" (OuterVolumeSpecName: "dshm") pod "576d14b8-3288-4dac-bbc8-d9ff8e2e9563" (UID: "576d14b8-3288-4dac-bbc8-d9ff8e2e9563"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:09:59.547665 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.547641    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "576d14b8-3288-4dac-bbc8-d9ff8e2e9563" (UID: "576d14b8-3288-4dac-bbc8-d9ff8e2e9563"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:09:59.547799 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.547665    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kube-api-access-z9g4n" (OuterVolumeSpecName: "kube-api-access-z9g4n") pod "576d14b8-3288-4dac-bbc8-d9ff8e2e9563" (UID: "576d14b8-3288-4dac-bbc8-d9ff8e2e9563"). InnerVolumeSpecName "kube-api-access-z9g4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:09:59.566167 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.566122    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "576d14b8-3288-4dac-bbc8-d9ff8e2e9563" (UID: "576d14b8-3288-4dac-bbc8-d9ff8e2e9563"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:09:59.616663 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.616622    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "576d14b8-3288-4dac-bbc8-d9ff8e2e9563" (UID: "576d14b8-3288-4dac-bbc8-d9ff8e2e9563"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:09:59.645731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.645693    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:09:59.645731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.645726    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:09:59.645731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.645736    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9g4n\" (UniqueName: \"kubernetes.io/projected/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-kube-api-access-z9g4n\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:09:59.645949 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.645746    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:09:59.645949 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.645754    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:09:59.645949 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.645761    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/576d14b8-3288-4dac-bbc8-d9ff8e2e9563-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:09:59.774595 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.774551    2543 generic.go:356] "Generic (PLEG): container finished" podID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerID="67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf" exitCode=0
Jul 07 03:09:59.774755 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.774599    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" event={"ID":"576d14b8-3288-4dac-bbc8-d9ff8e2e9563","Type":"ContainerDied","Data":"67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf"}
Jul 07 03:09:59.774755 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.774654    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" event={"ID":"576d14b8-3288-4dac-bbc8-d9ff8e2e9563","Type":"ContainerDied","Data":"970f2b599c76279ebbe6fafa0f49c3a16228eae76148366c6b347853a017cc83"}
Jul 07 03:09:59.774755 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.774609    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"
Jul 07 03:09:59.774755 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.774686    2543 scope.go:117] "RemoveContainer" containerID="67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf"
Jul 07 03:09:59.784549 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.784529    2543 scope.go:117] "RemoveContainer" containerID="335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e"
Jul 07 03:09:59.793453 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.793434    2543 scope.go:117] "RemoveContainer" containerID="67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf"
Jul 07 03:09:59.793758 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:09:59.793736    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf\": container with ID starting with 67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf not found: ID does not exist" containerID="67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf"
Jul 07 03:09:59.793847 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.793770    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf"} err="failed to get container status \"67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf\": rpc error: code = NotFound desc = could not find container \"67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf\": container with ID starting with 67f47f1c47e374d5c4e1a8e4a8fcf1d404943aa2262a4a3778e7833729e8ccbf not found: ID does not exist"
Jul 07 03:09:59.793847 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.793793    2543 scope.go:117] "RemoveContainer" containerID="335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e"
Jul 07 03:09:59.794055 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:09:59.794040    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e\": container with ID starting with 335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e not found: ID does not exist" containerID="335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e"
Jul 07 03:09:59.794117 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.794067    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e"} err="failed to get container status \"335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e\": rpc error: code = NotFound desc = could not find container \"335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e\": container with ID starting with 335f67a1acc05daac44823f0392e3bea0ff052d82c845fc34340a0f0d490c45e not found: ID does not exist"
Jul 07 03:09:59.801143 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.801087    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"]
Jul 07 03:09:59.804486 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.804463    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr"]
Jul 07 03:09:59.982201 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:09:59.982159    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" path="/var/lib/kubelet/pods/576d14b8-3288-4dac-bbc8-d9ff8e2e9563/volumes"
Jul 07 03:10:00.310025 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:00.309978    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-66dc875567-8hbzr" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded"
Jul 07 03:10:07.794233 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:07.794145    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:10:07.814880 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:07.814843    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:10:09.255246 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.255208    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"]
Jul 07 03:10:09.255636 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.255558    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="storage-initializer"
Jul 07 03:10:09.255636 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.255569    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="storage-initializer"
Jul 07 03:10:09.255636 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.255585    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main"
Jul 07 03:10:09.255636 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.255591    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main"
Jul 07 03:10:09.255766 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.255646    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="576d14b8-3288-4dac-bbc8-d9ff8e2e9563" containerName="main"
Jul 07 03:10:09.259304 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.259286    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.261813 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.261779    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 07 03:10:09.269241 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.269213    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"]
Jul 07 03:10:09.337259 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.337221    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8khm\" (UniqueName: \"kubernetes.io/projected/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kube-api-access-m8khm\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.337463 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.337285    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tmp-dir\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.337463 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.337322    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-dshm\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.337463 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.337344    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tls-certs\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.337463 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.337361    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-model-cache\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.337463 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.337383    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.337710 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.337491    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-home\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.438554 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.438519    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-home\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.438732 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.438596    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8khm\" (UniqueName: \"kubernetes.io/projected/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kube-api-access-m8khm\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.438732 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.438630    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tmp-dir\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.438732 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.438651    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-dshm\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.438732 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.438680    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tls-certs\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.438732 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.438706    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-model-cache\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.439012 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.438981    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-home\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.439177 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.439145    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-model-cache\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.439177 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.439055    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tmp-dir\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.439547 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.439033    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.439547 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.439303    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.441103 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.441077    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-dshm\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.441361 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.441345    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tls-certs\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.447053 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.447027    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8khm\" (UniqueName: \"kubernetes.io/projected/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kube-api-access-m8khm\") pod \"prestop-hook-test-kserve-7f76488d87-7g5z8\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.570060 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.569955    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:09.713643 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.713618    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"]
Jul 07 03:10:09.715827 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:10:09.715800    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8264cdf1_bbc2_4707_8c5a_c427ebdc8b62.slice/crio-c87feac5c1d209407456fad8c2057e04d18a2cef6a17a0c9eb769d416686cef0 WatchSource:0}: Error finding container c87feac5c1d209407456fad8c2057e04d18a2cef6a17a0c9eb769d416686cef0: Status 404 returned error can't find the container with id c87feac5c1d209407456fad8c2057e04d18a2cef6a17a0c9eb769d416686cef0
Jul 07 03:10:09.830683 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.830593    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" event={"ID":"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62","Type":"ContainerStarted","Data":"5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e"}
Jul 07 03:10:09.830683 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:09.830653    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" event={"ID":"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62","Type":"ContainerStarted","Data":"c87feac5c1d209407456fad8c2057e04d18a2cef6a17a0c9eb769d416686cef0"}
Jul 07 03:10:13.857156 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:13.857115    2543 generic.go:356] "Generic (PLEG): container finished" podID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerID="5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e" exitCode=0
Jul 07 03:10:13.857589 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:13.857205    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" event={"ID":"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62","Type":"ContainerDied","Data":"5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e"}
Jul 07 03:10:14.865822 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:14.865779    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" event={"ID":"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62","Type":"ContainerStarted","Data":"9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d"}
Jul 07 03:10:14.889977 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:14.889874    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podStartSLOduration=2.5108346729999997 podStartE2EDuration="5.889857266s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.379022593s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:10:14.889583847 +0000 UTC m=+3127.532744377" watchObservedRunningTime="2026-07-07 03:10:14.889857266 +0000 UTC m=+3127.533017788"
Jul 07 03:10:15.570569 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:15.570528    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:17.794072 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:17.794008    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:10:17.814470 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:17.814418    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:10:19.570663 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:19.570623    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:10:19.572374 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:19.572342    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:10:27.793625 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:27.793512    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:10:27.814366 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:27.814311    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:10:29.570921 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:29.570870    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:10:37.794293 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:37.794235    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:10:37.814331 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:37.814145    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:10:39.571246 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:39.571173    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:10:47.794295 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:47.794236    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:10:47.814047 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:47.814003    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:10:49.571358 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:49.571305    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:10:57.793786 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:57.793731    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:10:57.814216 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:57.814160    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:10:59.571326 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:10:59.571272    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:11:07.793805 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:07.793740    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:11:07.814406 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:07.814365    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:11:09.571323 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:09.571269    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:11:17.793885 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:17.793833    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:11:17.813947 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:17.813895    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:11:19.570436 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:19.570388    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:11:27.793883 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:27.793812    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8001/health\": dial tcp 10.134.0.58:8001: connect: connection refused"
Jul 07 03:11:27.814442 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:27.814381    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": dial tcp 10.134.0.59:8000: connect: connection refused"
Jul 07 03:11:29.571167 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:29.571121    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:11:37.804272 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:37.804243    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:11:37.822770 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:37.822746    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:11:37.827729 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:37.827706    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:11:37.837176 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:37.837146    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:11:39.570639 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:39.570581    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:11:49.571213 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:49.571129    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:11:53.626020 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:53.625981    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"]
Jul 07 03:11:53.630787 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:53.630754    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"]
Jul 07 03:11:59.571413 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:11:59.571301    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:12:08.656160 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:08.656117    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main" containerID="cri-o://ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc" gracePeriod=45
Jul 07 03:12:08.663335 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:08.663291    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main" containerID="cri-o://de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158" gracePeriod=45
Jul 07 03:12:09.570988 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.570946    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:12:09.783505 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.783458    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="llm-d-routing-sidecar" containerID="cri-o://4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7" gracePeriod=44
Jul 07 03:12:09.929969 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.929940    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:12:09.971012 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.970980    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-tmp-dir\") pod \"beeace29-430c-4b69-8a70-9bdcd069b691\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") "
Jul 07 03:12:09.971198 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.971034    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-model-cache\") pod \"beeace29-430c-4b69-8a70-9bdcd069b691\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") "
Jul 07 03:12:09.971198 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.971057    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-home\") pod \"beeace29-430c-4b69-8a70-9bdcd069b691\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") "
Jul 07 03:12:09.971198 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.971126    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xjx8\" (UniqueName: \"kubernetes.io/projected/beeace29-430c-4b69-8a70-9bdcd069b691-kube-api-access-7xjx8\") pod \"beeace29-430c-4b69-8a70-9bdcd069b691\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") "
Jul 07 03:12:09.971385 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.971261    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-kserve-provision-location\") pod \"beeace29-430c-4b69-8a70-9bdcd069b691\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") "
Jul 07 03:12:09.971385 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.971286    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-dshm\") pod \"beeace29-430c-4b69-8a70-9bdcd069b691\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") "
Jul 07 03:12:09.971385 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.971326    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beeace29-430c-4b69-8a70-9bdcd069b691-tls-certs\") pod \"beeace29-430c-4b69-8a70-9bdcd069b691\" (UID: \"beeace29-430c-4b69-8a70-9bdcd069b691\") "
Jul 07 03:12:09.971385 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.971335    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-model-cache" (OuterVolumeSpecName: "model-cache") pod "beeace29-430c-4b69-8a70-9bdcd069b691" (UID: "beeace29-430c-4b69-8a70-9bdcd069b691"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:09.971603 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.971583    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:09.972470 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.972427    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-home" (OuterVolumeSpecName: "home") pod "beeace29-430c-4b69-8a70-9bdcd069b691" (UID: "beeace29-430c-4b69-8a70-9bdcd069b691"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:09.976113 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.976070    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beeace29-430c-4b69-8a70-9bdcd069b691-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "beeace29-430c-4b69-8a70-9bdcd069b691" (UID: "beeace29-430c-4b69-8a70-9bdcd069b691"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:12:09.978238 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.978171    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beeace29-430c-4b69-8a70-9bdcd069b691-kube-api-access-7xjx8" (OuterVolumeSpecName: "kube-api-access-7xjx8") pod "beeace29-430c-4b69-8a70-9bdcd069b691" (UID: "beeace29-430c-4b69-8a70-9bdcd069b691"). InnerVolumeSpecName "kube-api-access-7xjx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:12:09.978545 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.978505    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-dshm" (OuterVolumeSpecName: "dshm") pod "beeace29-430c-4b69-8a70-9bdcd069b691" (UID: "beeace29-430c-4b69-8a70-9bdcd069b691"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:09.989165 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:09.989134    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "beeace29-430c-4b69-8a70-9bdcd069b691" (UID: "beeace29-430c-4b69-8a70-9bdcd069b691"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:10.034922 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.034877    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "beeace29-430c-4b69-8a70-9bdcd069b691" (UID: "beeace29-430c-4b69-8a70-9bdcd069b691"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:10.039436 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.039414    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:12:10.072542 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.072507    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8gf4\" (UniqueName: \"kubernetes.io/projected/7d82ee77-9347-440d-a18b-f958769e3180-kube-api-access-d8gf4\") pod \"7d82ee77-9347-440d-a18b-f958769e3180\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") "
Jul 07 03:12:10.072542 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.072547    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-model-cache\") pod \"7d82ee77-9347-440d-a18b-f958769e3180\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") "
Jul 07 03:12:10.072804 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.072571    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-tmp-dir\") pod \"7d82ee77-9347-440d-a18b-f958769e3180\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") "
Jul 07 03:12:10.072804 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.072599    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-dshm\") pod \"7d82ee77-9347-440d-a18b-f958769e3180\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") "
Jul 07 03:12:10.072804 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.072647    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d82ee77-9347-440d-a18b-f958769e3180-tls-certs\") pod \"7d82ee77-9347-440d-a18b-f958769e3180\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") "
Jul 07 03:12:10.072804 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.072680    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-home\") pod \"7d82ee77-9347-440d-a18b-f958769e3180\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") "
Jul 07 03:12:10.072804 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.072749    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-kserve-provision-location\") pod \"7d82ee77-9347-440d-a18b-f958769e3180\" (UID: \"7d82ee77-9347-440d-a18b-f958769e3180\") "
Jul 07 03:12:10.073082 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.072855    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-model-cache" (OuterVolumeSpecName: "model-cache") pod "7d82ee77-9347-440d-a18b-f958769e3180" (UID: "7d82ee77-9347-440d-a18b-f958769e3180"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:10.073082 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.073041    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xjx8\" (UniqueName: \"kubernetes.io/projected/beeace29-430c-4b69-8a70-9bdcd069b691-kube-api-access-7xjx8\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.073082 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.073063    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.073082 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.073078    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.073330 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.073094    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beeace29-430c-4b69-8a70-9bdcd069b691-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.073330 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.073108    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.073330 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.073121    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.073330 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.073135    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beeace29-430c-4b69-8a70-9bdcd069b691-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.074078 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.074046    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-home" (OuterVolumeSpecName: "home") pod "7d82ee77-9347-440d-a18b-f958769e3180" (UID: "7d82ee77-9347-440d-a18b-f958769e3180"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:10.075221 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.075162    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d82ee77-9347-440d-a18b-f958769e3180-kube-api-access-d8gf4" (OuterVolumeSpecName: "kube-api-access-d8gf4") pod "7d82ee77-9347-440d-a18b-f958769e3180" (UID: "7d82ee77-9347-440d-a18b-f958769e3180"). InnerVolumeSpecName "kube-api-access-d8gf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:12:10.075966 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.075933    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-dshm" (OuterVolumeSpecName: "dshm") pod "7d82ee77-9347-440d-a18b-f958769e3180" (UID: "7d82ee77-9347-440d-a18b-f958769e3180"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:10.076088 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.075967    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d82ee77-9347-440d-a18b-f958769e3180-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7d82ee77-9347-440d-a18b-f958769e3180" (UID: "7d82ee77-9347-440d-a18b-f958769e3180"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:12:10.090626 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.090536    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7d82ee77-9347-440d-a18b-f958769e3180" (UID: "7d82ee77-9347-440d-a18b-f958769e3180"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:10.113389 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.113338    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d82ee77-9347-440d-a18b-f958769e3180" (UID: "7d82ee77-9347-440d-a18b-f958769e3180"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:10.174356 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.174323    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8gf4\" (UniqueName: \"kubernetes.io/projected/7d82ee77-9347-440d-a18b-f958769e3180-kube-api-access-d8gf4\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.174356 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.174354    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.174356 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.174365    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.174600 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.174373    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d82ee77-9347-440d-a18b-f958769e3180-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.174600 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.174383    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.174600 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.174391    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d82ee77-9347-440d-a18b-f958769e3180-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:10.470525 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.470488    2543 generic.go:356] "Generic (PLEG): container finished" podID="7d82ee77-9347-440d-a18b-f958769e3180" containerID="ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc" exitCode=0
Jul 07 03:12:10.470525 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.470517    2543 generic.go:356] "Generic (PLEG): container finished" podID="7d82ee77-9347-440d-a18b-f958769e3180" containerID="4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7" exitCode=0
Jul 07 03:12:10.470797 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.470558    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"
Jul 07 03:12:10.470797 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.470580    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" event={"ID":"7d82ee77-9347-440d-a18b-f958769e3180","Type":"ContainerDied","Data":"ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc"}
Jul 07 03:12:10.470797 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.470640    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" event={"ID":"7d82ee77-9347-440d-a18b-f958769e3180","Type":"ContainerDied","Data":"4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7"}
Jul 07 03:12:10.470797 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.470688    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc" event={"ID":"7d82ee77-9347-440d-a18b-f958769e3180","Type":"ContainerDied","Data":"6fe9c98b1ccdac0fc73f41f0fe4c353588be01d5b088219736f23242a70c2368"}
Jul 07 03:12:10.470797 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.470717    2543 scope.go:117] "RemoveContainer" containerID="ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc"
Jul 07 03:12:10.472587 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.472559    2543 generic.go:356] "Generic (PLEG): container finished" podID="beeace29-430c-4b69-8a70-9bdcd069b691" containerID="de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158" exitCode=0
Jul 07 03:12:10.472727 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.472618    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"
Jul 07 03:12:10.472727 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.472621    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" event={"ID":"beeace29-430c-4b69-8a70-9bdcd069b691","Type":"ContainerDied","Data":"de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158"}
Jul 07 03:12:10.472727 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.472689    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r" event={"ID":"beeace29-430c-4b69-8a70-9bdcd069b691","Type":"ContainerDied","Data":"9e6a59aa0c8723c4ca55c870cc1de5651f99377518356985d4d33491bbe0af2a"}
Jul 07 03:12:10.490509 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.490489    2543 scope.go:117] "RemoveContainer" containerID="56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b"
Jul 07 03:12:10.499093 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.499075    2543 scope.go:117] "RemoveContainer" containerID="4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7"
Jul 07 03:12:10.503102 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.503079    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"]
Jul 07 03:12:10.509363 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.509340    2543 scope.go:117] "RemoveContainer" containerID="ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc"
Jul 07 03:12:10.509734 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.509712    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-84bd675bc9-kzc4r"]
Jul 07 03:12:10.509734 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:12:10.509718    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc\": container with ID starting with ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc not found: ID does not exist" containerID="ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc"
Jul 07 03:12:10.509846 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.509757    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc"} err="failed to get container status \"ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc\": rpc error: code = NotFound desc = could not find container \"ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc\": container with ID starting with ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc not found: ID does not exist"
Jul 07 03:12:10.509846 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.509785    2543 scope.go:117] "RemoveContainer" containerID="56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b"
Jul 07 03:12:10.510097 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:12:10.510080    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b\": container with ID starting with 56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b not found: ID does not exist" containerID="56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b"
Jul 07 03:12:10.510144 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.510103    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b"} err="failed to get container status \"56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b\": rpc error: code = NotFound desc = could not find container \"56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b\": container with ID starting with 56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b not found: ID does not exist"
Jul 07 03:12:10.510144 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.510126    2543 scope.go:117] "RemoveContainer" containerID="4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7"
Jul 07 03:12:10.510425 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:12:10.510409    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7\": container with ID starting with 4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7 not found: ID does not exist" containerID="4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7"
Jul 07 03:12:10.510485 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.510431    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7"} err="failed to get container status \"4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7\": rpc error: code = NotFound desc = could not find container \"4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7\": container with ID starting with 4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7 not found: ID does not exist"
Jul 07 03:12:10.510485 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.510446    2543 scope.go:117] "RemoveContainer" containerID="ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc"
Jul 07 03:12:10.510718 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.510693    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc"} err="failed to get container status \"ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc\": rpc error: code = NotFound desc = could not find container \"ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc\": container with ID starting with ca42849a66056f097521afe96ac14259a5f6cb2f2fbf8a69297d73e4cf60c6cc not found: ID does not exist"
Jul 07 03:12:10.510773 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.510722    2543 scope.go:117] "RemoveContainer" containerID="56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b"
Jul 07 03:12:10.510983 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.510962    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b"} err="failed to get container status \"56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b\": rpc error: code = NotFound desc = could not find container \"56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b\": container with ID starting with 56c6db324c8e172f6696d504619a0b42cdc18543faf9dec9ede70e4d98e5bd9b not found: ID does not exist"
Jul 07 03:12:10.511033 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.510984    2543 scope.go:117] "RemoveContainer" containerID="4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7"
Jul 07 03:12:10.511262 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.511245    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7"} err="failed to get container status \"4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7\": rpc error: code = NotFound desc = could not find container \"4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7\": container with ID starting with 4d9f7b862d98bdec6b882ae2ea2b6dd55c24b10f2056511e68635a7ade78eaa7 not found: ID does not exist"
Jul 07 03:12:10.511333 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.511263    2543 scope.go:117] "RemoveContainer" containerID="de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158"
Jul 07 03:12:10.520613 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.520593    2543 scope.go:117] "RemoveContainer" containerID="4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b"
Jul 07 03:12:10.526994 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.526969    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"]
Jul 07 03:12:10.530008 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.529976    2543 scope.go:117] "RemoveContainer" containerID="de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158"
Jul 07 03:12:10.530332 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:12:10.530312    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158\": container with ID starting with de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158 not found: ID does not exist" containerID="de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158"
Jul 07 03:12:10.530427 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.530339    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158"} err="failed to get container status \"de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158\": rpc error: code = NotFound desc = could not find container \"de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158\": container with ID starting with de098811ab33b053e25c842d29342de08ad53a26def4350506a6b4a1ff62e158 not found: ID does not exist"
Jul 07 03:12:10.530427 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.530358    2543 scope.go:117] "RemoveContainer" containerID="4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b"
Jul 07 03:12:10.530642 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:12:10.530625    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b\": container with ID starting with 4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b not found: ID does not exist" containerID="4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b"
Jul 07 03:12:10.530692 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.530646    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b"} err="failed to get container status \"4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b\": rpc error: code = NotFound desc = could not find container \"4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b\": container with ID starting with 4136e53be6c4aa19de447d7505dcd5fa14804fea408ac6e95e29d5928a1a221b not found: ID does not exist"
Jul 07 03:12:10.531924 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:10.531906    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-6f67d5cf8b-522cc"]
Jul 07 03:12:11.980580 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:11.980544    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d82ee77-9347-440d-a18b-f958769e3180" path="/var/lib/kubelet/pods/7d82ee77-9347-440d-a18b-f958769e3180/volumes"
Jul 07 03:12:11.981231 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:11.981214    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" path="/var/lib/kubelet/pods/beeace29-430c-4b69-8a70-9bdcd069b691/volumes"
Jul 07 03:12:16.506299 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506266    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"]
Jul 07 03:12:16.506689 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506620    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main"
Jul 07 03:12:16.506689 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506631    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main"
Jul 07 03:12:16.506689 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506642    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="storage-initializer"
Jul 07 03:12:16.506689 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506648    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="storage-initializer"
Jul 07 03:12:16.506689 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506657    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="llm-d-routing-sidecar"
Jul 07 03:12:16.506689 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506663    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="llm-d-routing-sidecar"
Jul 07 03:12:16.506689 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506673    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="storage-initializer"
Jul 07 03:12:16.506689 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506681    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="storage-initializer"
Jul 07 03:12:16.506936 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506697    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main"
Jul 07 03:12:16.506936 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506703    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main"
Jul 07 03:12:16.506936 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506756    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="main"
Jul 07 03:12:16.506936 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506769    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d82ee77-9347-440d-a18b-f958769e3180" containerName="llm-d-routing-sidecar"
Jul 07 03:12:16.506936 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.506776    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="beeace29-430c-4b69-8a70-9bdcd069b691" containerName="main"
Jul 07 03:12:16.511044 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.511023    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.513452 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.513424    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 07 03:12:16.513598 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.513519    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-wlc7g\""
Jul 07 03:12:16.523760 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.523733    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"]
Jul 07 03:12:16.635210 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.635142    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.635386 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.635224    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.635386 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.635244    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.635386 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.635265    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.635386 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.635287    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.635557 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.635390    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ede82e-7e84-438e-81df-f2ff0deb2257-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.635557 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.635449    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.635557 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.635505    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.635684 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.635571    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x74xd\" (UniqueName: \"kubernetes.io/projected/b5ede82e-7e84-438e-81df-f2ff0deb2257-kube-api-access-x74xd\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736244 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736201    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736442 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736263    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736442 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736307    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x74xd\" (UniqueName: \"kubernetes.io/projected/b5ede82e-7e84-438e-81df-f2ff0deb2257-kube-api-access-x74xd\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736442 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736339    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736442 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736400    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736442 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736426    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736713 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736466    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736713 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736497    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736713 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736525    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ede82e-7e84-438e-81df-f2ff0deb2257-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736871 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736824    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.736964 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.736936    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.737097 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.737070    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.737213 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.737118    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.737505 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.737481    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5ede82e-7e84-438e-81df-f2ff0deb2257-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.739020 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.739000    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.739120 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.739054    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.744434 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.744402    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5ede82e-7e84-438e-81df-f2ff0deb2257-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.744548 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.744531    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x74xd\" (UniqueName: \"kubernetes.io/projected/b5ede82e-7e84-438e-81df-f2ff0deb2257-kube-api-access-x74xd\") pod \"router-gateway-2-openshift-default-684dc9b4b9-bt7g2\" (UID: \"b5ede82e-7e84-438e-81df-f2ff0deb2257\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.822834 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.822742    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:16.992098 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.992073    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"]
Jul 07 03:12:16.995470 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:12:16.995433    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ede82e_7e84_438e_81df_f2ff0deb2257.slice/crio-31a398f70f69beb0cefcfb3dd67b0f087fa6b32a61d2d8f6eea3b5c6f29b2a33 WatchSource:0}: Error finding container 31a398f70f69beb0cefcfb3dd67b0f087fa6b32a61d2d8f6eea3b5c6f29b2a33: Status 404 returned error can't find the container with id 31a398f70f69beb0cefcfb3dd67b0f087fa6b32a61d2d8f6eea3b5c6f29b2a33
Jul 07 03:12:16.999294 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.998352    2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 07 03:12:16.999294 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.998456    2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 07 03:12:16.999294 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:16.998513    2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 07 03:12:17.508494 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:17.508450    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2" event={"ID":"b5ede82e-7e84-438e-81df-f2ff0deb2257","Type":"ContainerStarted","Data":"4e302577eaabce871cc0cc8288a7887ce6828465193ed2557edbb0ea159a24fa"}
Jul 07 03:12:17.509048 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:17.508526    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2" event={"ID":"b5ede82e-7e84-438e-81df-f2ff0deb2257","Type":"ContainerStarted","Data":"31a398f70f69beb0cefcfb3dd67b0f087fa6b32a61d2d8f6eea3b5c6f29b2a33"}
Jul 07 03:12:17.530003 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:17.529950    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2" podStartSLOduration=1.529931664 podStartE2EDuration="1.529931664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:12:17.528543166 +0000 UTC m=+3250.171703695" watchObservedRunningTime="2026-07-07 03:12:17.529931664 +0000 UTC m=+3250.173092186"
Jul 07 03:12:17.823755 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:17.823629    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:18.825306 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:18.825268    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2" podUID="b5ede82e-7e84-438e-81df-f2ff0deb2257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 07 03:12:19.571435 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:19.571389    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": dial tcp 10.134.0.60:8000: connect: connection refused"
Jul 07 03:12:19.823652 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:19.823555    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2" podUID="b5ede82e-7e84-438e-81df-f2ff0deb2257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 07 03:12:20.824152 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:20.824105    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2" podUID="b5ede82e-7e84-438e-81df-f2ff0deb2257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 07 03:12:21.823572 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:21.823529    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2" podUID="b5ede82e-7e84-438e-81df-f2ff0deb2257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 07 03:12:22.823634 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:22.823587    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2" podUID="b5ede82e-7e84-438e-81df-f2ff0deb2257" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 07 03:12:23.827591 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:23.827553    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:23.828093 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:23.827953    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:23.828719 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:23.828698    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-bt7g2"
Jul 07 03:12:29.488049 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.488012    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"]
Jul 07 03:12:29.490808 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.490791    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.493667 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.493647    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 07 03:12:29.493813 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.493648    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-z9xtx\""
Jul 07 03:12:29.511915 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.511883    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"]
Jul 07 03:12:29.537252 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.537215    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"]
Jul 07 03:12:29.539578 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.539564    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.555748 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.555718    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"]
Jul 07 03:12:29.558446 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.558421    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-model-cache\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.558572 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.558458    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.558572 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.558527    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.558572 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.558562    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k44k2\" (UniqueName: \"kubernetes.io/projected/45aeda78-a80b-405b-b20b-b9ab080d25a9-kube-api-access-k44k2\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.558716 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.558611    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45aeda78-a80b-405b-b20b-b9ab080d25a9-tls-certs\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.558716 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.558700    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-home\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.558812 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.558734    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-dshm\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.581402 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.581372    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:12:29.589834 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.589804    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:12:29.660021 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.659985    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-dshm\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.660239 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660029    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.660239 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660058    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.660239 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660078    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts8rs\" (UniqueName: \"kubernetes.io/projected/58a1a075-3cd7-43cf-8fe3-423916903f50-kube-api-access-ts8rs\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.660239 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660163    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-model-cache\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.660239 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660212    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.660518 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660251    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-home\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.660518 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660274    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.660518 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660316    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a1a075-3cd7-43cf-8fe3-423916903f50-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.660518 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660346    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.660518 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660370    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k44k2\" (UniqueName: \"kubernetes.io/projected/45aeda78-a80b-405b-b20b-b9ab080d25a9-kube-api-access-k44k2\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.660756 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660611    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-model-cache\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.660756 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660637    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45aeda78-a80b-405b-b20b-b9ab080d25a9-tls-certs\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.660756 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660652    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-tmp-dir\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.660756 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660684    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.660756 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660691    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.660949 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.660768    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-home\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.661055 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.661032    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-home\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.662714 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.662688    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-dshm\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.663070 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.663052    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45aeda78-a80b-405b-b20b-b9ab080d25a9-tls-certs\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.669261 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.669225    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k44k2\" (UniqueName: \"kubernetes.io/projected/45aeda78-a80b-405b-b20b-b9ab080d25a9-kube-api-access-k44k2\") pod \"router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.761512 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761411    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-home\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.761512 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761445    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.761512 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761462    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a1a075-3cd7-43cf-8fe3-423916903f50-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.761512 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761493    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.761821 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761523    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.761821 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761548    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.761821 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761567    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts8rs\" (UniqueName: \"kubernetes.io/projected/58a1a075-3cd7-43cf-8fe3-423916903f50-kube-api-access-ts8rs\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.761968 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761835    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-home\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.761968 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761948    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.762080 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761976    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.762080 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.761986    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.763993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.763973    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.764109 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.764042    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a1a075-3cd7-43cf-8fe3-423916903f50-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.770816 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.770792    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts8rs\" (UniqueName: \"kubernetes.io/projected/58a1a075-3cd7-43cf-8fe3-423916903f50-kube-api-access-ts8rs\") pod \"router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.800622 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.800584    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:29.851686 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.851599    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:29.955260 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:29.955229    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"]
Jul 07 03:12:29.957280 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:12:29.957248    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45aeda78_a80b_405b_b20b_b9ab080d25a9.slice/crio-ae9d7678ba93253ad2b01d75b053d8422488bcafeb69c209fc3dc53ea664cd9c WatchSource:0}: Error finding container ae9d7678ba93253ad2b01d75b053d8422488bcafeb69c209fc3dc53ea664cd9c: Status 404 returned error can't find the container with id ae9d7678ba93253ad2b01d75b053d8422488bcafeb69c209fc3dc53ea664cd9c
Jul 07 03:12:30.001996 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:30.001972    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"]
Jul 07 03:12:30.007230 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:12:30.007178    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a1a075_3cd7_43cf_8fe3_423916903f50.slice/crio-cb0f60b1d561ea3f0836c310a00193d1d268e2e89b9c6ad78d84ecc9d5a0a7df WatchSource:0}: Error finding container cb0f60b1d561ea3f0836c310a00193d1d268e2e89b9c6ad78d84ecc9d5a0a7df: Status 404 returned error can't find the container with id cb0f60b1d561ea3f0836c310a00193d1d268e2e89b9c6ad78d84ecc9d5a0a7df
Jul 07 03:12:30.582075 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:30.582027    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" event={"ID":"45aeda78-a80b-405b-b20b-b9ab080d25a9","Type":"ContainerStarted","Data":"18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3"}
Jul 07 03:12:30.582799 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:30.582648    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:30.583047 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:30.583029    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" event={"ID":"45aeda78-a80b-405b-b20b-b9ab080d25a9","Type":"ContainerStarted","Data":"ae9d7678ba93253ad2b01d75b053d8422488bcafeb69c209fc3dc53ea664cd9c"}
Jul 07 03:12:30.584770 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:30.584741    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" event={"ID":"58a1a075-3cd7-43cf-8fe3-423916903f50","Type":"ContainerStarted","Data":"cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505"}
Jul 07 03:12:30.584875 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:30.584792    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" event={"ID":"58a1a075-3cd7-43cf-8fe3-423916903f50","Type":"ContainerStarted","Data":"cb0f60b1d561ea3f0836c310a00193d1d268e2e89b9c6ad78d84ecc9d5a0a7df"}
Jul 07 03:12:31.593425 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:31.593173    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" event={"ID":"45aeda78-a80b-405b-b20b-b9ab080d25a9","Type":"ContainerStarted","Data":"8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4"}
Jul 07 03:12:34.611446 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.611414    2543 generic.go:356] "Generic (PLEG): container finished" podID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerID="8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4" exitCode=0
Jul 07 03:12:34.611913 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.611497    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" event={"ID":"45aeda78-a80b-405b-b20b-b9ab080d25a9","Type":"ContainerDied","Data":"8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4"}
Jul 07 03:12:34.612993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.612970    2543 generic.go:356] "Generic (PLEG): container finished" podID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerID="cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505" exitCode=0
Jul 07 03:12:34.613094 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.613044    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" event={"ID":"58a1a075-3cd7-43cf-8fe3-423916903f50","Type":"ContainerDied","Data":"cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505"}
Jul 07 03:12:34.685267 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.685172    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"]
Jul 07 03:12:34.709350 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.709315    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"]
Jul 07 03:12:34.712576 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.712555    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.728886 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.728842    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"]
Jul 07 03:12:34.809865 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.809815    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-model-cache\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.809865 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.809865    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-tmp-dir\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.810105 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.809907    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4n5\" (UniqueName: \"kubernetes.io/projected/31e0189f-6ce1-46e1-82cf-683e84215003-kube-api-access-2m4n5\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.810105 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.810002    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-home\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.810105 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.810037    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-dshm\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.810302 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.810121    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31e0189f-6ce1-46e1-82cf-683e84215003-tls-certs\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.810302 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.810233    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.911593 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.911544    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-home\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.911593 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.911598    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-dshm\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.911869 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.911645    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31e0189f-6ce1-46e1-82cf-683e84215003-tls-certs\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.911869 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.911712    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.911869 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.911803    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-model-cache\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.911869 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.911828    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-tmp-dir\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.911869 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.911862    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4n5\" (UniqueName: \"kubernetes.io/projected/31e0189f-6ce1-46e1-82cf-683e84215003-kube-api-access-2m4n5\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.912121 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.912024    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-home\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.912121 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.912085    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-kserve-provision-location\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.912276 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.912254    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-model-cache\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.912339 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.912306    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-tmp-dir\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.914362 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.914329    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-dshm\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.914662 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.914638    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31e0189f-6ce1-46e1-82cf-683e84215003-tls-certs\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:34.920609 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:34.920577    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4n5\" (UniqueName: \"kubernetes.io/projected/31e0189f-6ce1-46e1-82cf-683e84215003-kube-api-access-2m4n5\") pod \"prestop-hook-test-kserve-7f76488d87-zl2xg\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:35.024986 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:35.024943    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:35.186360 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:35.186331    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"]
Jul 07 03:12:35.189393 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:12:35.189359    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e0189f_6ce1_46e1_82cf_683e84215003.slice/crio-a82b06eee59eecb51931759e4e87fced9a5004268a4ded0c3b008af3c6950df8 WatchSource:0}: Error finding container a82b06eee59eecb51931759e4e87fced9a5004268a4ded0c3b008af3c6950df8: Status 404 returned error can't find the container with id a82b06eee59eecb51931759e4e87fced9a5004268a4ded0c3b008af3c6950df8
Jul 07 03:12:35.622736 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:35.622683    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg" event={"ID":"31e0189f-6ce1-46e1-82cf-683e84215003","Type":"ContainerStarted","Data":"1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859"}
Jul 07 03:12:35.623301 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:35.622754    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg" event={"ID":"31e0189f-6ce1-46e1-82cf-683e84215003","Type":"ContainerStarted","Data":"a82b06eee59eecb51931759e4e87fced9a5004268a4ded0c3b008af3c6950df8"}
Jul 07 03:12:35.625798 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:35.625765    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" event={"ID":"45aeda78-a80b-405b-b20b-b9ab080d25a9","Type":"ContainerStarted","Data":"0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158"}
Jul 07 03:12:35.628767 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:35.628738    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" event={"ID":"58a1a075-3cd7-43cf-8fe3-423916903f50","Type":"ContainerStarted","Data":"a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0"}
Jul 07 03:12:35.666991 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:35.665901    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podStartSLOduration=3.119626576 podStartE2EDuration="6.665880583s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.546254007s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:12:35.66444057 +0000 UTC m=+3268.307601095" watchObservedRunningTime="2026-07-07 03:12:35.665880583 +0000 UTC m=+3268.309041106"
Jul 07 03:12:35.686451 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:35.686382    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podStartSLOduration=3.413478211 podStartE2EDuration="6.686362635s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.272884424s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:12:35.685284465 +0000 UTC m=+3268.328445055" watchObservedRunningTime="2026-07-07 03:12:35.686362635 +0000 UTC m=+3268.329523158"
Jul 07 03:12:38.662486 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:38.662451    2543 generic.go:356] "Generic (PLEG): container finished" podID="31e0189f-6ce1-46e1-82cf-683e84215003" containerID="1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859" exitCode=0
Jul 07 03:12:38.662900 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:38.662482    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg" event={"ID":"31e0189f-6ce1-46e1-82cf-683e84215003","Type":"ContainerDied","Data":"1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859"}
Jul 07 03:12:39.671479 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:39.671433    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg" event={"ID":"31e0189f-6ce1-46e1-82cf-683e84215003","Type":"ContainerStarted","Data":"06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3"}
Jul 07 03:12:39.694787 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:39.694723    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg" podStartSLOduration=2.334828154 podStartE2EDuration="5.694701736s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.359873582s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:12:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:12:39.692695578 +0000 UTC m=+3272.335856101" watchObservedRunningTime="2026-07-07 03:12:39.694701736 +0000 UTC m=+3272.337862258"
Jul 07 03:12:39.800726 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:39.800688    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:39.801001 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:39.800801    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:39.802371 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:39.802339    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:12:39.852480 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:39.852443    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:39.852676 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:39.852524    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:12:39.854365 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:39.854332    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:12:40.025443 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:40.025346    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:45.026020 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:45.025979    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:12:45.027795 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:45.027759    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg" podUID="31e0189f-6ce1-46e1-82cf-683e84215003" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": dial tcp 10.134.0.64:8000: connect: connection refused"
Jul 07 03:12:49.710579 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:49.710530    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" containerID="cri-o://9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d" gracePeriod=45
Jul 07 03:12:49.801668 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:49.801612    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:12:49.823900 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:49.823868    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:12:49.852117 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:49.852082    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:12:50.643715 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.643630    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:12:50.681975 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.681868    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-home\") pod \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") "
Jul 07 03:12:50.681975 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.681908    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tls-certs\") pod \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") "
Jul 07 03:12:50.681975 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.681954    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kserve-provision-location\") pod \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") "
Jul 07 03:12:50.682315 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.681995    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8khm\" (UniqueName: \"kubernetes.io/projected/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kube-api-access-m8khm\") pod \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") "
Jul 07 03:12:50.682315 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.682039    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-dshm\") pod \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") "
Jul 07 03:12:50.682315 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.682070    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tmp-dir\") pod \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") "
Jul 07 03:12:50.682315 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.682108    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-model-cache\") pod \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\" (UID: \"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62\") "
Jul 07 03:12:50.683045 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.682538    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-home" (OuterVolumeSpecName: "home") pod "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" (UID: "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:50.683045 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.682742    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-model-cache" (OuterVolumeSpecName: "model-cache") pod "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" (UID: "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:50.706282 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.699681    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-dshm" (OuterVolumeSpecName: "dshm") pod "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" (UID: "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:50.706282 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.701672    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" (UID: "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:12:50.706755 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.706717    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kube-api-access-m8khm" (OuterVolumeSpecName: "kube-api-access-m8khm") pod "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" (UID: "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62"). InnerVolumeSpecName "kube-api-access-m8khm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:12:50.712799 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.712745    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" (UID: "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:50.761399 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.761358    2543 generic.go:356] "Generic (PLEG): container finished" podID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerID="9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d" exitCode=0
Jul 07 03:12:50.761612 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.761427    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"
Jul 07 03:12:50.761612 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.761476    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" event={"ID":"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62","Type":"ContainerDied","Data":"9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d"}
Jul 07 03:12:50.761612 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.761525    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" event={"ID":"8264cdf1-bbc2-4707-8c5a-c427ebdc8b62","Type":"ContainerDied","Data":"c87feac5c1d209407456fad8c2057e04d18a2cef6a17a0c9eb769d416686cef0"}
Jul 07 03:12:50.761612 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.761554    2543 scope.go:117] "RemoveContainer" containerID="9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d"
Jul 07 03:12:50.767077 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.767047    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" (UID: "8264cdf1-bbc2-4707-8c5a-c427ebdc8b62"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:12:50.773262 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.773205    2543 scope.go:117] "RemoveContainer" containerID="5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e"
Jul 07 03:12:50.783737 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.783712    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:50.783845 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.783746    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:50.783845 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.783762    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:50.783845 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.783776    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8khm\" (UniqueName: \"kubernetes.io/projected/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-kube-api-access-m8khm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:50.783845 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.783793    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:50.783845 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.783806    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:50.783845 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.783821    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:12:50.784429 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.784408    2543 scope.go:117] "RemoveContainer" containerID="9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d"
Jul 07 03:12:50.784769 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:12:50.784737    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d\": container with ID starting with 9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d not found: ID does not exist" containerID="9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d"
Jul 07 03:12:50.784856 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.784779    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d"} err="failed to get container status \"9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d\": rpc error: code = NotFound desc = could not find container \"9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d\": container with ID starting with 9e5cfabcd0525a8d0055fcfa76e7d3f0aa43c9d5bd67f704d67cbdeb220e763d not found: ID does not exist"
Jul 07 03:12:50.784856 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.784803    2543 scope.go:117] "RemoveContainer" containerID="5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e"
Jul 07 03:12:50.785128 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:12:50.785093    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e\": container with ID starting with 5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e not found: ID does not exist" containerID="5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e"
Jul 07 03:12:50.785235 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:50.785127    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e"} err="failed to get container status \"5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e\": rpc error: code = NotFound desc = could not find container \"5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e\": container with ID starting with 5678176ad912b50419f6c9f02007f9eacb769cfe07e90954b33105e778da8b2e not found: ID does not exist"
Jul 07 03:12:51.090306 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:51.090269    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"]
Jul 07 03:12:51.092979 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:51.092952    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8"]
Jul 07 03:12:51.582771 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:51.582721    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-7g5z8" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 07 03:12:51.981608 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:51.981562    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" path="/var/lib/kubelet/pods/8264cdf1-bbc2-4707-8c5a-c427ebdc8b62/volumes"
Jul 07 03:12:52.987777 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:52.987743    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"]
Jul 07 03:12:59.801475 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:59.801428    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:12:59.853031 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:12:59.852982    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:13:08.016492 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:08.016428    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg" podUID="31e0189f-6ce1-46e1-82cf-683e84215003" containerName="main" containerID="cri-o://06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3" gracePeriod=45
Jul 07 03:13:08.113164 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:08.113121    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:13:08.120095 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:08.120059    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:13:09.802900 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:09.802842    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:13:09.852718 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:09.852670    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:13:19.801953 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:19.801906    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:13:19.852942 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:19.852888    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:13:29.801518 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:29.801408    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:13:29.852441 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:29.852394    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:13:39.801120 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:39.801068    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:13:39.852473 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:39.852422    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:13:49.801414 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:49.801361    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:13:49.852813 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:49.852759    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:13:53.669088 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.669066    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-7f76488d87-zl2xg_31e0189f-6ce1-46e1-82cf-683e84215003/main/0.log"
Jul 07 03:13:53.671241 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.671221    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:13:53.706644 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.706610    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-tmp-dir\") pod \"31e0189f-6ce1-46e1-82cf-683e84215003\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") "
Jul 07 03:13:53.706833 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.706671    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m4n5\" (UniqueName: \"kubernetes.io/projected/31e0189f-6ce1-46e1-82cf-683e84215003-kube-api-access-2m4n5\") pod \"31e0189f-6ce1-46e1-82cf-683e84215003\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") "
Jul 07 03:13:53.706833 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.706720    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-home\") pod \"31e0189f-6ce1-46e1-82cf-683e84215003\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") "
Jul 07 03:13:53.706833 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.706747    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-model-cache\") pod \"31e0189f-6ce1-46e1-82cf-683e84215003\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") "
Jul 07 03:13:53.706833 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.706824    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-dshm\") pod \"31e0189f-6ce1-46e1-82cf-683e84215003\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") "
Jul 07 03:13:53.707056 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.706862    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31e0189f-6ce1-46e1-82cf-683e84215003-tls-certs\") pod \"31e0189f-6ce1-46e1-82cf-683e84215003\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") "
Jul 07 03:13:53.707056 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.706897    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-kserve-provision-location\") pod \"31e0189f-6ce1-46e1-82cf-683e84215003\" (UID: \"31e0189f-6ce1-46e1-82cf-683e84215003\") "
Jul 07 03:13:53.707056 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.707030    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31e0189f-6ce1-46e1-82cf-683e84215003" (UID: "31e0189f-6ce1-46e1-82cf-683e84215003"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:13:53.707056 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.707030    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-model-cache" (OuterVolumeSpecName: "model-cache") pod "31e0189f-6ce1-46e1-82cf-683e84215003" (UID: "31e0189f-6ce1-46e1-82cf-683e84215003"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:13:53.707316 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.707111    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-home" (OuterVolumeSpecName: "home") pod "31e0189f-6ce1-46e1-82cf-683e84215003" (UID: "31e0189f-6ce1-46e1-82cf-683e84215003"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:13:53.707316 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.707224    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:13:53.707316 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.707244    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:13:53.707316 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.707256    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:13:53.709091 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.709069    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31e0189f-6ce1-46e1-82cf-683e84215003-kube-api-access-2m4n5" (OuterVolumeSpecName: "kube-api-access-2m4n5") pod "31e0189f-6ce1-46e1-82cf-683e84215003" (UID: "31e0189f-6ce1-46e1-82cf-683e84215003"). InnerVolumeSpecName "kube-api-access-2m4n5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:13:53.709288 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.709269    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-dshm" (OuterVolumeSpecName: "dshm") pod "31e0189f-6ce1-46e1-82cf-683e84215003" (UID: "31e0189f-6ce1-46e1-82cf-683e84215003"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:13:53.710487 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.710469    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31e0189f-6ce1-46e1-82cf-683e84215003-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31e0189f-6ce1-46e1-82cf-683e84215003" (UID: "31e0189f-6ce1-46e1-82cf-683e84215003"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:13:53.773206 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.773157    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "31e0189f-6ce1-46e1-82cf-683e84215003" (UID: "31e0189f-6ce1-46e1-82cf-683e84215003"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:13:53.808550 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.808458    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:13:53.808550 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.808492    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31e0189f-6ce1-46e1-82cf-683e84215003-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:13:53.808550 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.808507    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/31e0189f-6ce1-46e1-82cf-683e84215003-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:13:53.808550 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:53.808519    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m4n5\" (UniqueName: \"kubernetes.io/projected/31e0189f-6ce1-46e1-82cf-683e84215003-kube-api-access-2m4n5\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:13:54.142373 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.142345    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-7f76488d87-zl2xg_31e0189f-6ce1-46e1-82cf-683e84215003/main/0.log"
Jul 07 03:13:54.142794 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.142766    2543 generic.go:356] "Generic (PLEG): container finished" podID="31e0189f-6ce1-46e1-82cf-683e84215003" containerID="06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3" exitCode=137
Jul 07 03:13:54.142871 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.142829    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"
Jul 07 03:13:54.142871 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.142852    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg" event={"ID":"31e0189f-6ce1-46e1-82cf-683e84215003","Type":"ContainerDied","Data":"06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3"}
Jul 07 03:13:54.142993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.142918    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg" event={"ID":"31e0189f-6ce1-46e1-82cf-683e84215003","Type":"ContainerDied","Data":"a82b06eee59eecb51931759e4e87fced9a5004268a4ded0c3b008af3c6950df8"}
Jul 07 03:13:54.142993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.142952    2543 scope.go:117] "RemoveContainer" containerID="06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3"
Jul 07 03:13:54.151833 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.151806    2543 scope.go:117] "RemoveContainer" containerID="1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859"
Jul 07 03:13:54.160089 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.160056    2543 scope.go:117] "RemoveContainer" containerID="06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3"
Jul 07 03:13:54.160544 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:13:54.160509    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3\": container with ID starting with 06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3 not found: ID does not exist" containerID="06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3"
Jul 07 03:13:54.160690 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.160551    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3"} err="failed to get container status \"06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3\": rpc error: code = NotFound desc = could not find container \"06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3\": container with ID starting with 06391ac84c703e6e7df7ea57794e8ef558c369335f2893bd2524eaf6f58f2ef3 not found: ID does not exist"
Jul 07 03:13:54.160690 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.160571    2543 scope.go:117] "RemoveContainer" containerID="1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859"
Jul 07 03:13:54.160904 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:13:54.160867    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859\": container with ID starting with 1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859 not found: ID does not exist" containerID="1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859"
Jul 07 03:13:54.160965 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.160913    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859"} err="failed to get container status \"1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859\": rpc error: code = NotFound desc = could not find container \"1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859\": container with ID starting with 1730ddabf9363fb91a355e99f8123457153a3371859e26f9dedc3f00c8a66859 not found: ID does not exist"
Jul 07 03:13:54.162441 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.162419    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"]
Jul 07 03:13:54.166735 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:54.166714    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-7f76488d87-zl2xg"]
Jul 07 03:13:55.980873 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:55.980837    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31e0189f-6ce1-46e1-82cf-683e84215003" path="/var/lib/kubelet/pods/31e0189f-6ce1-46e1-82cf-683e84215003/volumes"
Jul 07 03:13:59.801604 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:59.801558    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:13:59.852205 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:13:59.852150    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:14:09.801339 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:09.801276    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:14:09.853027 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:09.852973    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:14:19.801953 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:19.801907    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:14:19.852232 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:19.852173    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:14:29.801974 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:29.801929    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:14:29.852780 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:29.852744    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:14:39.801163 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:39.801115    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:14:39.852592 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:39.852550    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:14:49.801982 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:49.801933    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:14:49.852618 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:49.852569    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:14:59.801797 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:59.801677    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:14:59.852885 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:14:59.852840    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:15:09.801561 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:09.801514    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8001/health\": dial tcp 10.134.0.62:8001: connect: connection refused"
Jul 07 03:15:09.853075 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:09.853028    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": dial tcp 10.134.0.63:8000: connect: connection refused"
Jul 07 03:15:19.811042 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:19.811012    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:15:19.823777 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:19.823755    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:15:19.864542 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:19.864516    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:15:19.874274 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:19.874251    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:15:28.536119 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:28.536088    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"]
Jul 07 03:15:28.540098 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:28.540067    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"]
Jul 07 03:15:43.560973 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:43.560926    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main" containerID="cri-o://a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0" gracePeriod=45
Jul 07 03:15:43.564577 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:43.564540    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main" containerID="cri-o://0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158" gracePeriod=45
Jul 07 03:15:44.380788 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.380754    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="llm-d-routing-sidecar" containerID="cri-o://18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3" gracePeriod=45
Jul 07 03:15:44.516012 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.515988    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:15:44.617081 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.617053    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-tmp-dir\") pod \"58a1a075-3cd7-43cf-8fe3-423916903f50\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") "
Jul 07 03:15:44.617601 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.617090    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a1a075-3cd7-43cf-8fe3-423916903f50-tls-certs\") pod \"58a1a075-3cd7-43cf-8fe3-423916903f50\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") "
Jul 07 03:15:44.617601 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.617114    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts8rs\" (UniqueName: \"kubernetes.io/projected/58a1a075-3cd7-43cf-8fe3-423916903f50-kube-api-access-ts8rs\") pod \"58a1a075-3cd7-43cf-8fe3-423916903f50\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") "
Jul 07 03:15:44.617601 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.617141    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-model-cache\") pod \"58a1a075-3cd7-43cf-8fe3-423916903f50\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") "
Jul 07 03:15:44.617601 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.617233    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-dshm\") pod \"58a1a075-3cd7-43cf-8fe3-423916903f50\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") "
Jul 07 03:15:44.617601 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.617267    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-home\") pod \"58a1a075-3cd7-43cf-8fe3-423916903f50\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") "
Jul 07 03:15:44.617601 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.617323    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-kserve-provision-location\") pod \"58a1a075-3cd7-43cf-8fe3-423916903f50\" (UID: \"58a1a075-3cd7-43cf-8fe3-423916903f50\") "
Jul 07 03:15:44.617601 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.617524    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-model-cache" (OuterVolumeSpecName: "model-cache") pod "58a1a075-3cd7-43cf-8fe3-423916903f50" (UID: "58a1a075-3cd7-43cf-8fe3-423916903f50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:44.618028 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.617758    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.618403 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.618372    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-home" (OuterVolumeSpecName: "home") pod "58a1a075-3cd7-43cf-8fe3-423916903f50" (UID: "58a1a075-3cd7-43cf-8fe3-423916903f50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:44.619796 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.619771    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a1a075-3cd7-43cf-8fe3-423916903f50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58a1a075-3cd7-43cf-8fe3-423916903f50" (UID: "58a1a075-3cd7-43cf-8fe3-423916903f50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:15:44.619936 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.619909    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-dshm" (OuterVolumeSpecName: "dshm") pod "58a1a075-3cd7-43cf-8fe3-423916903f50" (UID: "58a1a075-3cd7-43cf-8fe3-423916903f50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:44.619994 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.619979    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a1a075-3cd7-43cf-8fe3-423916903f50-kube-api-access-ts8rs" (OuterVolumeSpecName: "kube-api-access-ts8rs") pod "58a1a075-3cd7-43cf-8fe3-423916903f50" (UID: "58a1a075-3cd7-43cf-8fe3-423916903f50"). InnerVolumeSpecName "kube-api-access-ts8rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:15:44.626775 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.626755    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:15:44.631170 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.631145    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58a1a075-3cd7-43cf-8fe3-423916903f50" (UID: "58a1a075-3cd7-43cf-8fe3-423916903f50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:44.680134 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.680099    2543 generic.go:356] "Generic (PLEG): container finished" podID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerID="a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0" exitCode=0
Jul 07 03:15:44.680334 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.680158    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"
Jul 07 03:15:44.680334 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.680204    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" event={"ID":"58a1a075-3cd7-43cf-8fe3-423916903f50","Type":"ContainerDied","Data":"a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0"}
Jul 07 03:15:44.680334 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.680269    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg" event={"ID":"58a1a075-3cd7-43cf-8fe3-423916903f50","Type":"ContainerDied","Data":"cb0f60b1d561ea3f0836c310a00193d1d268e2e89b9c6ad78d84ecc9d5a0a7df"}
Jul 07 03:15:44.680334 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.680299    2543 scope.go:117] "RemoveContainer" containerID="a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0"
Jul 07 03:15:44.682700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.682679    2543 generic.go:356] "Generic (PLEG): container finished" podID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerID="0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158" exitCode=0
Jul 07 03:15:44.682700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.682698    2543 generic.go:356] "Generic (PLEG): container finished" podID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerID="18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3" exitCode=0
Jul 07 03:15:44.682862 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.682740    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" event={"ID":"45aeda78-a80b-405b-b20b-b9ab080d25a9","Type":"ContainerDied","Data":"0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158"}
Jul 07 03:15:44.682862 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.682755    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"
Jul 07 03:15:44.682862 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.682785    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" event={"ID":"45aeda78-a80b-405b-b20b-b9ab080d25a9","Type":"ContainerDied","Data":"18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3"}
Jul 07 03:15:44.682862 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.682808    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk" event={"ID":"45aeda78-a80b-405b-b20b-b9ab080d25a9","Type":"ContainerDied","Data":"ae9d7678ba93253ad2b01d75b053d8422488bcafeb69c209fc3dc53ea664cd9c"}
Jul 07 03:15:44.689118 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.689100    2543 scope.go:117] "RemoveContainer" containerID="cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505"
Jul 07 03:15:44.698699 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.698671    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58a1a075-3cd7-43cf-8fe3-423916903f50" (UID: "58a1a075-3cd7-43cf-8fe3-423916903f50"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:44.698878 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.698863    2543 scope.go:117] "RemoveContainer" containerID="a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0"
Jul 07 03:15:44.699130 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:15:44.699112    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0\": container with ID starting with a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0 not found: ID does not exist" containerID="a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0"
Jul 07 03:15:44.699233 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.699150    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0"} err="failed to get container status \"a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0\": rpc error: code = NotFound desc = could not find container \"a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0\": container with ID starting with a6e6c07072058f39a450f23208f7077212d93ae3cf82d6202cdbc603e62010c0 not found: ID does not exist"
Jul 07 03:15:44.699233 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.699174    2543 scope.go:117] "RemoveContainer" containerID="cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505"
Jul 07 03:15:44.699442 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:15:44.699426    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505\": container with ID starting with cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505 not found: ID does not exist" containerID="cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505"
Jul 07 03:15:44.699505 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.699449    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505"} err="failed to get container status \"cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505\": rpc error: code = NotFound desc = could not find container \"cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505\": container with ID starting with cffbd5380fd99276881094f6ca10d85b73f532172d111ff59a3d4ddcc2ac3505 not found: ID does not exist"
Jul 07 03:15:44.699505 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.699470    2543 scope.go:117] "RemoveContainer" containerID="0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158"
Jul 07 03:15:44.706927 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.706909    2543 scope.go:117] "RemoveContainer" containerID="8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4"
Jul 07 03:15:44.714517 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.714501    2543 scope.go:117] "RemoveContainer" containerID="18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3"
Jul 07 03:15:44.718765 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.718743    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k44k2\" (UniqueName: \"kubernetes.io/projected/45aeda78-a80b-405b-b20b-b9ab080d25a9-kube-api-access-k44k2\") pod \"45aeda78-a80b-405b-b20b-b9ab080d25a9\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") "
Jul 07 03:15:44.718859 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.718787    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-model-cache\") pod \"45aeda78-a80b-405b-b20b-b9ab080d25a9\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") "
Jul 07 03:15:44.718915 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.718872    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-dshm\") pod \"45aeda78-a80b-405b-b20b-b9ab080d25a9\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") "
Jul 07 03:15:44.718915 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.718908    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45aeda78-a80b-405b-b20b-b9ab080d25a9-tls-certs\") pod \"45aeda78-a80b-405b-b20b-b9ab080d25a9\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") "
Jul 07 03:15:44.719018 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.718937    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-home\") pod \"45aeda78-a80b-405b-b20b-b9ab080d25a9\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") "
Jul 07 03:15:44.719018 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.718959    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-tmp-dir\") pod \"45aeda78-a80b-405b-b20b-b9ab080d25a9\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") "
Jul 07 03:15:44.719018 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.718994    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-kserve-provision-location\") pod \"45aeda78-a80b-405b-b20b-b9ab080d25a9\" (UID: \"45aeda78-a80b-405b-b20b-b9ab080d25a9\") "
Jul 07 03:15:44.719214 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.719123    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-model-cache" (OuterVolumeSpecName: "model-cache") pod "45aeda78-a80b-405b-b20b-b9ab080d25a9" (UID: "45aeda78-a80b-405b-b20b-b9ab080d25a9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:44.719382 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.719365    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.719441 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.719389    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.719441 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.719405    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.719441 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.719419    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.719441 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.719431    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58a1a075-3cd7-43cf-8fe3-423916903f50-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.719655 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.719445    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts8rs\" (UniqueName: \"kubernetes.io/projected/58a1a075-3cd7-43cf-8fe3-423916903f50-kube-api-access-ts8rs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.719655 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.719513    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58a1a075-3cd7-43cf-8fe3-423916903f50-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.719993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.719957    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-home" (OuterVolumeSpecName: "home") pod "45aeda78-a80b-405b-b20b-b9ab080d25a9" (UID: "45aeda78-a80b-405b-b20b-b9ab080d25a9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:44.721043 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.721023    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45aeda78-a80b-405b-b20b-b9ab080d25a9-kube-api-access-k44k2" (OuterVolumeSpecName: "kube-api-access-k44k2") pod "45aeda78-a80b-405b-b20b-b9ab080d25a9" (UID: "45aeda78-a80b-405b-b20b-b9ab080d25a9"). InnerVolumeSpecName "kube-api-access-k44k2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:15:44.721332 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.721309    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45aeda78-a80b-405b-b20b-b9ab080d25a9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45aeda78-a80b-405b-b20b-b9ab080d25a9" (UID: "45aeda78-a80b-405b-b20b-b9ab080d25a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:15:44.721508 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.721486    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-dshm" (OuterVolumeSpecName: "dshm") pod "45aeda78-a80b-405b-b20b-b9ab080d25a9" (UID: "45aeda78-a80b-405b-b20b-b9ab080d25a9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:44.722740 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.722720    2543 scope.go:117] "RemoveContainer" containerID="0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158"
Jul 07 03:15:44.723043 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:15:44.723011    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158\": container with ID starting with 0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158 not found: ID does not exist" containerID="0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158"
Jul 07 03:15:44.723120 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.723047    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158"} err="failed to get container status \"0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158\": rpc error: code = NotFound desc = could not find container \"0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158\": container with ID starting with 0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158 not found: ID does not exist"
Jul 07 03:15:44.723120 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.723064    2543 scope.go:117] "RemoveContainer" containerID="8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4"
Jul 07 03:15:44.723361 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:15:44.723336    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4\": container with ID starting with 8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4 not found: ID does not exist" containerID="8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4"
Jul 07 03:15:44.723403 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.723365    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4"} err="failed to get container status \"8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4\": rpc error: code = NotFound desc = could not find container \"8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4\": container with ID starting with 8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4 not found: ID does not exist"
Jul 07 03:15:44.723403 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.723381    2543 scope.go:117] "RemoveContainer" containerID="18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3"
Jul 07 03:15:44.723603 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:15:44.723586    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3\": container with ID starting with 18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3 not found: ID does not exist" containerID="18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3"
Jul 07 03:15:44.723641 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.723609    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3"} err="failed to get container status \"18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3\": rpc error: code = NotFound desc = could not find container \"18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3\": container with ID starting with 18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3 not found: ID does not exist"
Jul 07 03:15:44.723641 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.723626    2543 scope.go:117] "RemoveContainer" containerID="0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158"
Jul 07 03:15:44.723860 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.723842    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158"} err="failed to get container status \"0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158\": rpc error: code = NotFound desc = could not find container \"0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158\": container with ID starting with 0c5b9f2cc1d6611133110e9cc34f7caa8270b462d560778bf62136138dd1c158 not found: ID does not exist"
Jul 07 03:15:44.723860 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.723860    2543 scope.go:117] "RemoveContainer" containerID="8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4"
Jul 07 03:15:44.724077 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.724060    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4"} err="failed to get container status \"8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4\": rpc error: code = NotFound desc = could not find container \"8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4\": container with ID starting with 8c0c55a23143e0eec4d3bc8343f181d4ae2c360f2b05deb4c14267cf92ffc2b4 not found: ID does not exist"
Jul 07 03:15:44.724129 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.724082    2543 scope.go:117] "RemoveContainer" containerID="18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3"
Jul 07 03:15:44.724324 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.724303    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3"} err="failed to get container status \"18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3\": rpc error: code = NotFound desc = could not find container \"18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3\": container with ID starting with 18a29cb3cc3225127d31f8c1110b0f5b0cb28ece0949f22d66964503179515f3 not found: ID does not exist"
Jul 07 03:15:44.740753 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.740733    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45aeda78-a80b-405b-b20b-b9ab080d25a9" (UID: "45aeda78-a80b-405b-b20b-b9ab080d25a9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:44.793789 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.793753    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45aeda78-a80b-405b-b20b-b9ab080d25a9" (UID: "45aeda78-a80b-405b-b20b-b9ab080d25a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:15:44.821090 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.821052    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k44k2\" (UniqueName: \"kubernetes.io/projected/45aeda78-a80b-405b-b20b-b9ab080d25a9-kube-api-access-k44k2\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.821090 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.821080    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.821090 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.821090    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45aeda78-a80b-405b-b20b-b9ab080d25a9-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.821090 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.821098    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.821367 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.821106    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:44.821367 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:44.821114    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45aeda78-a80b-405b-b20b-b9ab080d25a9-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:15:45.006571 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:45.006544    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"]
Jul 07 03:15:45.010221 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:45.010177    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-69db96b599-2qzbg"]
Jul 07 03:15:45.023349 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:45.023326    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"]
Jul 07 03:15:45.027780 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:45.027758    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-5fdcf4bcfd-jcdfk"]
Jul 07 03:15:45.980555 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:45.980523    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" path="/var/lib/kubelet/pods/45aeda78-a80b-405b-b20b-b9ab080d25a9/volumes"
Jul 07 03:15:45.980990 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:15:45.980976    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" path="/var/lib/kubelet/pods/58a1a075-3cd7-43cf-8fe3-423916903f50/volumes"
Jul 07 03:16:18.661827 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.661791    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"]
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662098    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e0189f-6ce1-46e1-82cf-683e84215003" containerName="storage-initializer"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662109    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e0189f-6ce1-46e1-82cf-683e84215003" containerName="storage-initializer"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662124    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="storage-initializer"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662129    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="storage-initializer"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662135    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="storage-initializer"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662142    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="storage-initializer"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662148    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662153    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662159    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662166    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662172    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="storage-initializer"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662178    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="storage-initializer"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662211    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662219    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662232    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="llm-d-routing-sidecar"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662240    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="llm-d-routing-sidecar"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662248    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31e0189f-6ce1-46e1-82cf-683e84215003" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662253    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="31e0189f-6ce1-46e1-82cf-683e84215003" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662304    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="llm-d-routing-sidecar"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662312    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="45aeda78-a80b-405b-b20b-b9ab080d25a9" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662318    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="31e0189f-6ce1-46e1-82cf-683e84215003" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662328    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="58a1a075-3cd7-43cf-8fe3-423916903f50" containerName="main"
Jul 07 03:16:18.662327 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.662333    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8264cdf1-bbc2-4707-8c5a-c427ebdc8b62" containerName="main"
Jul 07 03:16:18.665005 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.664987    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.667900 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.667876    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 03:16:18.668016 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.667950    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 07 03:16:18.668016 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.667989    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 03:16:18.675902 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.675878    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"]
Jul 07 03:16:18.695807 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.695777    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h2nm\" (UniqueName: \"kubernetes.io/projected/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kube-api-access-2h2nm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.695937 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.695918    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.696013 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.695997    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.696096 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.696050    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.696161 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.696119    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.696313 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.696274    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.696434 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.696344    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.796834 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.796802    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.796834 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.796837    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.797084 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.796864    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.797084 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.796933    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.797084 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.796981    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.797084 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.797040    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h2nm\" (UniqueName: \"kubernetes.io/projected/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kube-api-access-2h2nm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.797084 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.797065    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.797468 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.797429    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.797559 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.797507    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.797652 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.797625    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.798005 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.797751    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.799640 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.799616    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.800085 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.800059    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.812164 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.812121    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h2nm\" (UniqueName: \"kubernetes.io/projected/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kube-api-access-2h2nm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:18.976012 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:18.975914    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:19.113770 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:19.113744    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"]
Jul 07 03:16:19.116409 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:16:19.116380    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdae6ef0_5dc9_458f_a856_7bb4ac4e7872.slice/crio-a8e38d79daf651b0dba64c37483b08e7335b2d92291a150b86c78cdb05b42141 WatchSource:0}: Error finding container a8e38d79daf651b0dba64c37483b08e7335b2d92291a150b86c78cdb05b42141: Status 404 returned error can't find the container with id a8e38d79daf651b0dba64c37483b08e7335b2d92291a150b86c78cdb05b42141
Jul 07 03:16:19.852419 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:19.852387    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" event={"ID":"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872","Type":"ContainerStarted","Data":"0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229"}
Jul 07 03:16:19.852810 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:19.852435    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" event={"ID":"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872","Type":"ContainerStarted","Data":"a8e38d79daf651b0dba64c37483b08e7335b2d92291a150b86c78cdb05b42141"}
Jul 07 03:16:24.879815 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:24.879781    2543 generic.go:356] "Generic (PLEG): container finished" podID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerID="0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229" exitCode=0
Jul 07 03:16:24.880219 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:24.879842    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" event={"ID":"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872","Type":"ContainerDied","Data":"0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229"}
Jul 07 03:16:25.886902 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:25.886864    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" event={"ID":"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872","Type":"ContainerStarted","Data":"7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67"}
Jul 07 03:16:25.909116 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:25.909058    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podStartSLOduration=2.341020807 podStartE2EDuration="7.909042355s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.568021548s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:16:25.907526884 +0000 UTC m=+3498.550687405" watchObservedRunningTime="2026-07-07 03:16:25.909042355 +0000 UTC m=+3498.552202879"
Jul 07 03:16:25.980883 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:25.980851    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:28.976369 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:28.976285    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:16:28.977795 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:28.977767    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:16:38.976384 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:38.976339    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:16:48.977172 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:48.977119    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:16:58.976591 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:16:58.976536    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:17:08.977055 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:08.977002    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:17:18.976540 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:18.976496    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:17:28.977237 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:28.977177    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:17:38.977370 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:38.977325    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:17:48.976368 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:48.976322    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:17:49.887411 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:49.887376    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"]
Jul 07 03:17:49.889992 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:49.889973    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:49.893388 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:49.893364    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-x6jt8\""
Jul 07 03:17:49.893514 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:49.893425    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 07 03:17:49.900142 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:49.900117    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"]
Jul 07 03:17:50.010700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.010649    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b16a413d-9a94-4c08-82d7-71495d45602e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:50.011088 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.010716    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dcrq\" (UniqueName: \"kubernetes.io/projected/b16a413d-9a94-4c08-82d7-71495d45602e-kube-api-access-8dcrq\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:50.011088 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.010827    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b16a413d-9a94-4c08-82d7-71495d45602e-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:50.111567 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.111529    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b16a413d-9a94-4c08-82d7-71495d45602e-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:50.111750 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.111614    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b16a413d-9a94-4c08-82d7-71495d45602e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:50.111750 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.111646    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dcrq\" (UniqueName: \"kubernetes.io/projected/b16a413d-9a94-4c08-82d7-71495d45602e-kube-api-access-8dcrq\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:50.112030 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.112005    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b16a413d-9a94-4c08-82d7-71495d45602e-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:50.114337 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.114317    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b16a413d-9a94-4c08-82d7-71495d45602e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:50.122165 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.122140    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dcrq\" (UniqueName: \"kubernetes.io/projected/b16a413d-9a94-4c08-82d7-71495d45602e-kube-api-access-8dcrq\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:50.200278 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.200182    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:50.350711 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:50.350680    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"]
Jul 07 03:17:50.354575 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:17:50.354546    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16a413d_9a94_4c08_82d7_71495d45602e.slice/crio-83e0bd857441d1687d1ed641a66472ae53eb4acc6f32080a5d872dbea48c8139 WatchSource:0}: Error finding container 83e0bd857441d1687d1ed641a66472ae53eb4acc6f32080a5d872dbea48c8139: Status 404 returned error can't find the container with id 83e0bd857441d1687d1ed641a66472ae53eb4acc6f32080a5d872dbea48c8139
Jul 07 03:17:51.283456 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:51.283419    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp" event={"ID":"b16a413d-9a94-4c08-82d7-71495d45602e","Type":"ContainerStarted","Data":"65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4"}
Jul 07 03:17:51.283982 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:51.283468    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp" event={"ID":"b16a413d-9a94-4c08-82d7-71495d45602e","Type":"ContainerStarted","Data":"83e0bd857441d1687d1ed641a66472ae53eb4acc6f32080a5d872dbea48c8139"}
Jul 07 03:17:51.283982 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:51.283547    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:17:51.300988 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:51.300939    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp" podStartSLOduration=2.300922811 podStartE2EDuration="2.300922811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:17:51.300126187 +0000 UTC m=+3583.943286709" watchObservedRunningTime="2026-07-07 03:17:51.300922811 +0000 UTC m=+3583.944083333"
Jul 07 03:17:58.976544 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:17:58.976438    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:18:08.147266 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:08.147233    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:18:08.155472 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:08.155444    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:18:08.977110 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:08.977068    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": dial tcp 10.134.0.65:8000: connect: connection refused"
Jul 07 03:18:18.986677 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:18.986645    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:18:18.994360 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:18.994337    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:18:22.291537 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:22.291502    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp" podUID="b16a413d-9a94-4c08-82d7-71495d45602e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 07 03:18:23.102439 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:23.102410    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"]
Jul 07 03:18:24.000568 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.000533    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"]
Jul 07 03:18:24.002259 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.002173    2543 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp" podUID="b16a413d-9a94-4c08-82d7-71495d45602e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 07 03:18:24.012077 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:18:24.012043    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:18:24.012077 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:18:24.012077 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 03:18:24.012077 ip-10-0-132-101 kubenswrapper[2543]:  > containerID="65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4" cmd=["/bin/sleep","15"]
Jul 07 03:18:24.012401 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.012097    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp" message="" err=<
Jul 07 03:18:24.012401 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:18:24.012401 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 03:18:24.012401 ip-10-0-132-101 kubenswrapper[2543]:  >
Jul 07 03:18:24.012401 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:18:24.012112    2543 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:18:24.012401 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:18:24.012401 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 03:18:24.012401 ip-10-0-132-101 kubenswrapper[2543]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp" podUID="b16a413d-9a94-4c08-82d7-71495d45602e" containerName="main" containerID="cri-o://65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4"
Jul 07 03:18:24.012401 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.012168    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp" podUID="b16a413d-9a94-4c08-82d7-71495d45602e" containerName="main" containerID="cri-o://65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4" gracePeriod=60
Jul 07 03:18:24.262198 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.262124    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:18:24.413291 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.413249    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b16a413d-9a94-4c08-82d7-71495d45602e-tls-certs\") pod \"b16a413d-9a94-4c08-82d7-71495d45602e\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") "
Jul 07 03:18:24.413495 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.413405    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dcrq\" (UniqueName: \"kubernetes.io/projected/b16a413d-9a94-4c08-82d7-71495d45602e-kube-api-access-8dcrq\") pod \"b16a413d-9a94-4c08-82d7-71495d45602e\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") "
Jul 07 03:18:24.413495 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.413438    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b16a413d-9a94-4c08-82d7-71495d45602e-tokenizer-uds\") pod \"b16a413d-9a94-4c08-82d7-71495d45602e\" (UID: \"b16a413d-9a94-4c08-82d7-71495d45602e\") "
Jul 07 03:18:24.413773 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.413747    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b16a413d-9a94-4c08-82d7-71495d45602e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "b16a413d-9a94-4c08-82d7-71495d45602e" (UID: "b16a413d-9a94-4c08-82d7-71495d45602e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:18:24.415584 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.415546    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b16a413d-9a94-4c08-82d7-71495d45602e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b16a413d-9a94-4c08-82d7-71495d45602e" (UID: "b16a413d-9a94-4c08-82d7-71495d45602e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:18:24.415691 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.415606    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16a413d-9a94-4c08-82d7-71495d45602e-kube-api-access-8dcrq" (OuterVolumeSpecName: "kube-api-access-8dcrq") pod "b16a413d-9a94-4c08-82d7-71495d45602e" (UID: "b16a413d-9a94-4c08-82d7-71495d45602e"). InnerVolumeSpecName "kube-api-access-8dcrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:18:24.444247 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.444182    2543 generic.go:356] "Generic (PLEG): container finished" podID="b16a413d-9a94-4c08-82d7-71495d45602e" containerID="65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4" exitCode=0
Jul 07 03:18:24.444405 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.444249    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"
Jul 07 03:18:24.444405 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.444273    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp" event={"ID":"b16a413d-9a94-4c08-82d7-71495d45602e","Type":"ContainerDied","Data":"65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4"}
Jul 07 03:18:24.444405 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.444319    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp" event={"ID":"b16a413d-9a94-4c08-82d7-71495d45602e","Type":"ContainerDied","Data":"83e0bd857441d1687d1ed641a66472ae53eb4acc6f32080a5d872dbea48c8139"}
Jul 07 03:18:24.444405 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.444339    2543 scope.go:117] "RemoveContainer" containerID="65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4"
Jul 07 03:18:24.460017 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.459980    2543 scope.go:117] "RemoveContainer" containerID="65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4"
Jul 07 03:18:24.460447 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:18:24.460420    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4\": container with ID starting with 65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4 not found: ID does not exist" containerID="65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4"
Jul 07 03:18:24.460547 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.460455    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4"} err="failed to get container status \"65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4\": rpc error: code = NotFound desc = could not find container \"65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4\": container with ID starting with 65b55ad46377417c8418f2368eb1849c01e75292e09dfc6e14ceb8ca34fe7cc4 not found: ID does not exist"
Jul 07 03:18:24.468496 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.468462    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"]
Jul 07 03:18:24.471421 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.471396    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85dhjphp"]
Jul 07 03:18:24.514131 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.514036    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dcrq\" (UniqueName: \"kubernetes.io/projected/b16a413d-9a94-4c08-82d7-71495d45602e-kube-api-access-8dcrq\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:18:24.514131 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.514069    2543 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/b16a413d-9a94-4c08-82d7-71495d45602e-tokenizer-uds\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:18:24.514131 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:24.514084    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b16a413d-9a94-4c08-82d7-71495d45602e-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:18:25.980051 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:25.980016    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b16a413d-9a94-4c08-82d7-71495d45602e" path="/var/lib/kubelet/pods/b16a413d-9a94-4c08-82d7-71495d45602e/volumes"
Jul 07 03:18:38.126318 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:38.126258    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main" containerID="cri-o://7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67" gracePeriod=45
Jul 07 03:18:38.888071 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:38.888047    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:18:39.040900 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.040803    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-home\") pod \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") "
Jul 07 03:18:39.040900 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.040852    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kserve-provision-location\") pod \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") "
Jul 07 03:18:39.040900 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.040876    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-dshm\") pod \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") "
Jul 07 03:18:39.041220 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.040912    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tls-certs\") pod \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") "
Jul 07 03:18:39.041220 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.040949    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-model-cache\") pod \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") "
Jul 07 03:18:39.041220 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.040975    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tmp-dir\") pod \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") "
Jul 07 03:18:39.041220 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.041037    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h2nm\" (UniqueName: \"kubernetes.io/projected/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kube-api-access-2h2nm\") pod \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\" (UID: \"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872\") "
Jul 07 03:18:39.041461 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.041226    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-model-cache" (OuterVolumeSpecName: "model-cache") pod "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" (UID: "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:18:39.041461 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.041318    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:18:39.041574 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.041548    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-home" (OuterVolumeSpecName: "home") pod "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" (UID: "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:18:39.043223 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.043146    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-dshm" (OuterVolumeSpecName: "dshm") pod "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" (UID: "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:18:39.043353 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.043244    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" (UID: "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:18:39.043524 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.043501    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kube-api-access-2h2nm" (OuterVolumeSpecName: "kube-api-access-2h2nm") pod "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" (UID: "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872"). InnerVolumeSpecName "kube-api-access-2h2nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:18:39.059337 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.059306    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" (UID: "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:18:39.121535 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.121478    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" (UID: "bdae6ef0-5dc9-458f-a856-7bb4ac4e7872"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:18:39.142090 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.142058    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:18:39.142090 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.142088    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:18:39.142498 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.142101    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:18:39.142498 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.142113    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:18:39.142498 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.142124    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:18:39.142498 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.142135    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h2nm\" (UniqueName: \"kubernetes.io/projected/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872-kube-api-access-2h2nm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:18:39.514935 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.514898    2543 generic.go:356] "Generic (PLEG): container finished" podID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerID="7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67" exitCode=0
Jul 07 03:18:39.515125 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.514956    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"
Jul 07 03:18:39.515125 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.514990    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" event={"ID":"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872","Type":"ContainerDied","Data":"7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67"}
Jul 07 03:18:39.515125 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.515049    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr" event={"ID":"bdae6ef0-5dc9-458f-a856-7bb4ac4e7872","Type":"ContainerDied","Data":"a8e38d79daf651b0dba64c37483b08e7335b2d92291a150b86c78cdb05b42141"}
Jul 07 03:18:39.515125 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.515073    2543 scope.go:117] "RemoveContainer" containerID="7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67"
Jul 07 03:18:39.524237 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.524219    2543 scope.go:117] "RemoveContainer" containerID="0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229"
Jul 07 03:18:39.532010 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.531995    2543 scope.go:117] "RemoveContainer" containerID="7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67"
Jul 07 03:18:39.532227 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:18:39.532203    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67\": container with ID starting with 7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67 not found: ID does not exist" containerID="7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67"
Jul 07 03:18:39.532334 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.532230    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67"} err="failed to get container status \"7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67\": rpc error: code = NotFound desc = could not find container \"7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67\": container with ID starting with 7d42b8a54fa3e75afbe9904f491ece65c4a0e0b7a2552f8328ac17a1a91f5e67 not found: ID does not exist"
Jul 07 03:18:39.532334 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.532246    2543 scope.go:117] "RemoveContainer" containerID="0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229"
Jul 07 03:18:39.532472 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:18:39.532454    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229\": container with ID starting with 0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229 not found: ID does not exist" containerID="0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229"
Jul 07 03:18:39.532523 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.532480    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229"} err="failed to get container status \"0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229\": rpc error: code = NotFound desc = could not find container \"0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229\": container with ID starting with 0de8d22d2ca0056dfeaf155ca3f2f37c6a99c395355fb684de46b151879fb229 not found: ID does not exist"
Jul 07 03:18:39.537267 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.537242    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"]
Jul 07 03:18:39.540857 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.540838    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-59675c7fb47xmlr"]
Jul 07 03:18:39.981357 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:39.981325    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" path="/var/lib/kubelet/pods/bdae6ef0-5dc9-458f-a856-7bb4ac4e7872/volumes"
Jul 07 03:18:58.152320 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.152285    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 07 03:18:58.152731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.152610    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main"
Jul 07 03:18:58.152731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.152621    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main"
Jul 07 03:18:58.152731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.152638    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b16a413d-9a94-4c08-82d7-71495d45602e" containerName="main"
Jul 07 03:18:58.152731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.152643    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b16a413d-9a94-4c08-82d7-71495d45602e" containerName="main"
Jul 07 03:18:58.152731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.152653    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="storage-initializer"
Jul 07 03:18:58.152731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.152660    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="storage-initializer"
Jul 07 03:18:58.152731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.152712    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="b16a413d-9a94-4c08-82d7-71495d45602e" containerName="main"
Jul 07 03:18:58.152731 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.152720    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdae6ef0-5dc9-458f-a856-7bb4ac4e7872" containerName="main"
Jul 07 03:18:58.155793 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.155773    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.159258 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.159238    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 07 03:18:58.159386 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.159256    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 03:18:58.159386 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.159265    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 07 03:18:58.159386 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.159321    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-5bpmd\""
Jul 07 03:18:58.167817 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.167791    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 07 03:18:58.191775 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.191750    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.191878 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.191780    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.191878 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.191864    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.191960 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.191904    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3038731a-e281-4b3f-a1db-aace2be0ddc5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.191997 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.191959    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.192045 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.192032    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.192082 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.192055    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxzkn\" (UniqueName: \"kubernetes.io/projected/3038731a-e281-4b3f-a1db-aace2be0ddc5-kube-api-access-rxzkn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.202580 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.202558    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 07 03:18:58.205867 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.205849    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.216947 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.216922    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 07 03:18:58.292832 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.292791    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.293103 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293074    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.293270 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293134    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183633a3-721d-488b-8a68-1812ea1b2621-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.293270 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293175    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4klgp\" (UniqueName: \"kubernetes.io/projected/183633a3-721d-488b-8a68-1812ea1b2621-kube-api-access-4klgp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.293270 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293177    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.293270 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293265    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.293519 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293297    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.293519 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293336    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.293519 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293356    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3038731a-e281-4b3f-a1db-aace2be0ddc5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.293519 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293391    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.293519 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293407    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.293772 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293545    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.293772 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293585    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.293772 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293624    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.293772 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293683    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.293772 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293716    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxzkn\" (UniqueName: \"kubernetes.io/projected/3038731a-e281-4b3f-a1db-aace2be0ddc5-kube-api-access-rxzkn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.293772 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293770    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.294075 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.293901    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.296008 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.295986    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.296343 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.296325    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3038731a-e281-4b3f-a1db-aace2be0ddc5-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.306933 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.306910    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxzkn\" (UniqueName: \"kubernetes.io/projected/3038731a-e281-4b3f-a1db-aace2be0ddc5-kube-api-access-rxzkn\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.394817 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.394783    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.395029 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.394854    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.395029 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.394878    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.395029 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.394917    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.395029 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.394952    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183633a3-721d-488b-8a68-1812ea1b2621-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.395029 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.394977    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4klgp\" (UniqueName: \"kubernetes.io/projected/183633a3-721d-488b-8a68-1812ea1b2621-kube-api-access-4klgp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.395029 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.395015    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.395394 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.395160    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.395394 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.395258    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.395394 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.395318    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.395558 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.395404    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.397258 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.397239    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.397427 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.397410    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183633a3-721d-488b-8a68-1812ea1b2621-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.403011 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.402954    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4klgp\" (UniqueName: \"kubernetes.io/projected/183633a3-721d-488b-8a68-1812ea1b2621-kube-api-access-4klgp\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.465924 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.465885    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:18:58.516096 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.516045    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:18:58.622647 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.622615    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 07 03:18:58.624496 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:18:58.624465    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3038731a_e281_4b3f_a1db_aace2be0ddc5.slice/crio-3a0378b7cbffc4e4815e4cb7b040edb30d8000bdaadaf633bf868c780846e8ac WatchSource:0}: Error finding container 3a0378b7cbffc4e4815e4cb7b040edb30d8000bdaadaf633bf868c780846e8ac: Status 404 returned error can't find the container with id 3a0378b7cbffc4e4815e4cb7b040edb30d8000bdaadaf633bf868c780846e8ac
Jul 07 03:18:58.672947 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:58.672923    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 07 03:18:58.675648 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:18:58.675621    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183633a3_721d_488b_8a68_1812ea1b2621.slice/crio-550e74e6347d8e6214fcdcdb2218fe5a93ef807fbae3fcaed2b7ef4cd61bbd5c WatchSource:0}: Error finding container 550e74e6347d8e6214fcdcdb2218fe5a93ef807fbae3fcaed2b7ef4cd61bbd5c: Status 404 returned error can't find the container with id 550e74e6347d8e6214fcdcdb2218fe5a93ef807fbae3fcaed2b7ef4cd61bbd5c
Jul 07 03:18:59.626842 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:59.626752    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"183633a3-721d-488b-8a68-1812ea1b2621","Type":"ContainerStarted","Data":"732260327fd6eb57fb2c8b22def5d2e7c4690d124cd6e524461fc9c68d962da7"}
Jul 07 03:18:59.627687 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:59.627294    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"183633a3-721d-488b-8a68-1812ea1b2621","Type":"ContainerStarted","Data":"550e74e6347d8e6214fcdcdb2218fe5a93ef807fbae3fcaed2b7ef4cd61bbd5c"}
Jul 07 03:18:59.631097 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:59.630998    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3038731a-e281-4b3f-a1db-aace2be0ddc5","Type":"ContainerStarted","Data":"dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac"}
Jul 07 03:18:59.632093 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:18:59.632053    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3038731a-e281-4b3f-a1db-aace2be0ddc5","Type":"ContainerStarted","Data":"3a0378b7cbffc4e4815e4cb7b040edb30d8000bdaadaf633bf868c780846e8ac"}
Jul 07 03:19:02.651778 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:02.651745    2543 generic.go:356] "Generic (PLEG): container finished" podID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerID="dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac" exitCode=0
Jul 07 03:19:02.652286 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:02.651786    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3038731a-e281-4b3f-a1db-aace2be0ddc5","Type":"ContainerDied","Data":"dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac"}
Jul 07 03:19:03.658388 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:03.658354    2543 generic.go:356] "Generic (PLEG): container finished" podID="183633a3-721d-488b-8a68-1812ea1b2621" containerID="732260327fd6eb57fb2c8b22def5d2e7c4690d124cd6e524461fc9c68d962da7" exitCode=0
Jul 07 03:19:03.658845 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:03.658431    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"183633a3-721d-488b-8a68-1812ea1b2621","Type":"ContainerDied","Data":"732260327fd6eb57fb2c8b22def5d2e7c4690d124cd6e524461fc9c68d962da7"}
Jul 07 03:19:03.660679 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:03.660656    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3038731a-e281-4b3f-a1db-aace2be0ddc5","Type":"ContainerStarted","Data":"cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95"}
Jul 07 03:19:03.701336 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:03.701283    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.30821419 podStartE2EDuration="5.701264525s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.393050335s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:19:03.697617212 +0000 UTC m=+3656.340777745" watchObservedRunningTime="2026-07-07 03:19:03.701264525 +0000 UTC m=+3656.344425073"
Jul 07 03:19:04.466805 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:04.466767    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:19:04.668629 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:04.668591    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"183633a3-721d-488b-8a68-1812ea1b2621","Type":"ContainerStarted","Data":"2b379e0dc9c019c1611e6f2edd87250ed8ba68319d5a1da2de8bddee9515adad"}
Jul 07 03:19:04.691903 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:04.691851    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.431879709 podStartE2EDuration="6.691830382s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.259950673s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:19:04.690008578 +0000 UTC m=+3657.333169100" watchObservedRunningTime="2026-07-07 03:19:04.691830382 +0000 UTC m=+3657.334990903"
Jul 07 03:19:08.466540 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:08.466499    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:19:08.468499 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:08.468464    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:19:18.467075 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:18.467025    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:19:28.466817 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:28.466710    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:19:38.466889 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:38.466841    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:19:48.467088 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:48.467040    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:19:58.467230 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:19:58.467160    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:20:08.466960 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:20:08.466912    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:20:18.467302 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:20:18.467260    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:20:28.467014 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:20:28.466972    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:20:38.467122 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:20:38.467067    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:20:48.466485 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:20:48.466439    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:20:58.466467 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:20:58.466368    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:21:08.466873 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:08.466828    2543 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.67:8000/health\": dial tcp 10.134.0.67:8000: connect: connection refused"
Jul 07 03:21:18.477227 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:18.477173    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:21:18.485396 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:18.485375    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:21:33.580377 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:33.580342    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 07 03:21:48.605133 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:48.605091    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main" containerID="cri-o://cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95" gracePeriod=45
Jul 07 03:21:49.364134 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.364103    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:21:49.492495 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.492401    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3038731a-e281-4b3f-a1db-aace2be0ddc5-tls-certs\") pod \"3038731a-e281-4b3f-a1db-aace2be0ddc5\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") "
Jul 07 03:21:49.492495 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.492467    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-dshm\") pod \"3038731a-e281-4b3f-a1db-aace2be0ddc5\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") "
Jul 07 03:21:49.492495 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.492491    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-kserve-provision-location\") pod \"3038731a-e281-4b3f-a1db-aace2be0ddc5\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") "
Jul 07 03:21:49.492846 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.492531    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-home\") pod \"3038731a-e281-4b3f-a1db-aace2be0ddc5\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") "
Jul 07 03:21:49.492846 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.492547    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxzkn\" (UniqueName: \"kubernetes.io/projected/3038731a-e281-4b3f-a1db-aace2be0ddc5-kube-api-access-rxzkn\") pod \"3038731a-e281-4b3f-a1db-aace2be0ddc5\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") "
Jul 07 03:21:49.492846 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.492574    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-model-cache\") pod \"3038731a-e281-4b3f-a1db-aace2be0ddc5\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") "
Jul 07 03:21:49.492846 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.492593    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-tmp-dir\") pod \"3038731a-e281-4b3f-a1db-aace2be0ddc5\" (UID: \"3038731a-e281-4b3f-a1db-aace2be0ddc5\") "
Jul 07 03:21:49.493071 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.492996    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-model-cache" (OuterVolumeSpecName: "model-cache") pod "3038731a-e281-4b3f-a1db-aace2be0ddc5" (UID: "3038731a-e281-4b3f-a1db-aace2be0ddc5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:49.493491 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.493460    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-home" (OuterVolumeSpecName: "home") pod "3038731a-e281-4b3f-a1db-aace2be0ddc5" (UID: "3038731a-e281-4b3f-a1db-aace2be0ddc5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:49.495169 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.495144    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-dshm" (OuterVolumeSpecName: "dshm") pod "3038731a-e281-4b3f-a1db-aace2be0ddc5" (UID: "3038731a-e281-4b3f-a1db-aace2be0ddc5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:49.495367 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.495307    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3038731a-e281-4b3f-a1db-aace2be0ddc5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3038731a-e281-4b3f-a1db-aace2be0ddc5" (UID: "3038731a-e281-4b3f-a1db-aace2be0ddc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:21:49.495367 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.495334    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3038731a-e281-4b3f-a1db-aace2be0ddc5-kube-api-access-rxzkn" (OuterVolumeSpecName: "kube-api-access-rxzkn") pod "3038731a-e281-4b3f-a1db-aace2be0ddc5" (UID: "3038731a-e281-4b3f-a1db-aace2be0ddc5"). InnerVolumeSpecName "kube-api-access-rxzkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:21:49.505531 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.505500    2543 generic.go:356] "Generic (PLEG): container finished" podID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerID="cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95" exitCode=0
Jul 07 03:21:49.505648 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.505566    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 07 03:21:49.505648 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.505586    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3038731a-e281-4b3f-a1db-aace2be0ddc5","Type":"ContainerDied","Data":"cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95"}
Jul 07 03:21:49.505723 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.505649    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"3038731a-e281-4b3f-a1db-aace2be0ddc5","Type":"ContainerDied","Data":"3a0378b7cbffc4e4815e4cb7b040edb30d8000bdaadaf633bf868c780846e8ac"}
Jul 07 03:21:49.505723 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.505679    2543 scope.go:117] "RemoveContainer" containerID="cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95"
Jul 07 03:21:49.506364 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.506333    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3038731a-e281-4b3f-a1db-aace2be0ddc5" (UID: "3038731a-e281-4b3f-a1db-aace2be0ddc5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:49.514452 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.514430    2543 scope.go:117] "RemoveContainer" containerID="dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac"
Jul 07 03:21:49.522953 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.522932    2543 scope.go:117] "RemoveContainer" containerID="cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95"
Jul 07 03:21:49.523435 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:21:49.523181    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95\": container with ID starting with cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95 not found: ID does not exist" containerID="cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95"
Jul 07 03:21:49.523512 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.523445    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95"} err="failed to get container status \"cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95\": rpc error: code = NotFound desc = could not find container \"cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95\": container with ID starting with cecf6a7ed7859af7d29ad2747ff8cb27fec0ac12e81131ff4636c105dc4a8a95 not found: ID does not exist"
Jul 07 03:21:49.523512 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.523467    2543 scope.go:117] "RemoveContainer" containerID="dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac"
Jul 07 03:21:49.523743 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:21:49.523719    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac\": container with ID starting with dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac not found: ID does not exist" containerID="dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac"
Jul 07 03:21:49.523790 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.523753    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac"} err="failed to get container status \"dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac\": rpc error: code = NotFound desc = could not find container \"dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac\": container with ID starting with dedcbcbee7f0d8f82b4468a67ad2c51874a757b2db6eabe132e8b535615eecac not found: ID does not exist"
Jul 07 03:21:49.560125 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.560090    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3038731a-e281-4b3f-a1db-aace2be0ddc5" (UID: "3038731a-e281-4b3f-a1db-aace2be0ddc5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:21:49.593602 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.593565    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:21:49.593602 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.593598    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxzkn\" (UniqueName: \"kubernetes.io/projected/3038731a-e281-4b3f-a1db-aace2be0ddc5-kube-api-access-rxzkn\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:21:49.593602 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.593610    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:21:49.593817 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.593619    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:21:49.593817 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.593627    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3038731a-e281-4b3f-a1db-aace2be0ddc5-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:21:49.593817 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.593634    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:21:49.593817 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.593643    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3038731a-e281-4b3f-a1db-aace2be0ddc5-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:21:49.827140 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.827101    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 07 03:21:49.830668 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.830632    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 07 03:21:49.845023 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.844983    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 07 03:21:49.980668 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:21:49.980633    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" path="/var/lib/kubelet/pods/3038731a-e281-4b3f-a1db-aace2be0ddc5/volumes"
Jul 07 03:22:04.869016 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:04.868973    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="183633a3-721d-488b-8a68-1812ea1b2621" containerName="main" containerID="cri-o://2b379e0dc9c019c1611e6f2edd87250ed8ba68319d5a1da2de8bddee9515adad" gracePeriod=45
Jul 07 03:22:05.584620 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.584587    2543 generic.go:356] "Generic (PLEG): container finished" podID="183633a3-721d-488b-8a68-1812ea1b2621" containerID="2b379e0dc9c019c1611e6f2edd87250ed8ba68319d5a1da2de8bddee9515adad" exitCode=0
Jul 07 03:22:05.584807 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.584667    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"183633a3-721d-488b-8a68-1812ea1b2621","Type":"ContainerDied","Data":"2b379e0dc9c019c1611e6f2edd87250ed8ba68319d5a1da2de8bddee9515adad"}
Jul 07 03:22:05.628664 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.628637    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:22:05.737299 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.737176    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4klgp\" (UniqueName: \"kubernetes.io/projected/183633a3-721d-488b-8a68-1812ea1b2621-kube-api-access-4klgp\") pod \"183633a3-721d-488b-8a68-1812ea1b2621\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") "
Jul 07 03:22:05.737299 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.737293    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-model-cache\") pod \"183633a3-721d-488b-8a68-1812ea1b2621\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") "
Jul 07 03:22:05.737598 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.737324    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183633a3-721d-488b-8a68-1812ea1b2621-tls-certs\") pod \"183633a3-721d-488b-8a68-1812ea1b2621\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") "
Jul 07 03:22:05.737598 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.737349    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-tmp-dir\") pod \"183633a3-721d-488b-8a68-1812ea1b2621\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") "
Jul 07 03:22:05.737598 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.737372    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-kserve-provision-location\") pod \"183633a3-721d-488b-8a68-1812ea1b2621\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") "
Jul 07 03:22:05.737598 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.737414    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-home\") pod \"183633a3-721d-488b-8a68-1812ea1b2621\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") "
Jul 07 03:22:05.737598 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.737429    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-dshm\") pod \"183633a3-721d-488b-8a68-1812ea1b2621\" (UID: \"183633a3-721d-488b-8a68-1812ea1b2621\") "
Jul 07 03:22:05.737876 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.737610    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-model-cache" (OuterVolumeSpecName: "model-cache") pod "183633a3-721d-488b-8a68-1812ea1b2621" (UID: "183633a3-721d-488b-8a68-1812ea1b2621"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:22:05.738107 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.738046    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-home" (OuterVolumeSpecName: "home") pod "183633a3-721d-488b-8a68-1812ea1b2621" (UID: "183633a3-721d-488b-8a68-1812ea1b2621"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:22:05.739807 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.739773    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/183633a3-721d-488b-8a68-1812ea1b2621-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "183633a3-721d-488b-8a68-1812ea1b2621" (UID: "183633a3-721d-488b-8a68-1812ea1b2621"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:22:05.740051 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.740018    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-dshm" (OuterVolumeSpecName: "dshm") pod "183633a3-721d-488b-8a68-1812ea1b2621" (UID: "183633a3-721d-488b-8a68-1812ea1b2621"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:22:05.740151 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.740089    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/183633a3-721d-488b-8a68-1812ea1b2621-kube-api-access-4klgp" (OuterVolumeSpecName: "kube-api-access-4klgp") pod "183633a3-721d-488b-8a68-1812ea1b2621" (UID: "183633a3-721d-488b-8a68-1812ea1b2621"). InnerVolumeSpecName "kube-api-access-4klgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:22:05.752503 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.752470    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "183633a3-721d-488b-8a68-1812ea1b2621" (UID: "183633a3-721d-488b-8a68-1812ea1b2621"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:22:05.803880 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.803834    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "183633a3-721d-488b-8a68-1812ea1b2621" (UID: "183633a3-721d-488b-8a68-1812ea1b2621"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:22:05.839070 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.839037    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:22:05.839070 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.839069    2543 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-kserve-provision-location\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:22:05.839230 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.839080    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:22:05.839230 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.839090    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:22:05.839230 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.839104    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4klgp\" (UniqueName: \"kubernetes.io/projected/183633a3-721d-488b-8a68-1812ea1b2621-kube-api-access-4klgp\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:22:05.839230 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.839114    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/183633a3-721d-488b-8a68-1812ea1b2621-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:22:05.839230 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:05.839121    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/183633a3-721d-488b-8a68-1812ea1b2621-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:22:06.591089 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:06.591047    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"183633a3-721d-488b-8a68-1812ea1b2621","Type":"ContainerDied","Data":"550e74e6347d8e6214fcdcdb2218fe5a93ef807fbae3fcaed2b7ef4cd61bbd5c"}
Jul 07 03:22:06.591546 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:06.591102    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 07 03:22:06.591546 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:06.591130    2543 scope.go:117] "RemoveContainer" containerID="2b379e0dc9c019c1611e6f2edd87250ed8ba68319d5a1da2de8bddee9515adad"
Jul 07 03:22:06.600145 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:06.600124    2543 scope.go:117] "RemoveContainer" containerID="732260327fd6eb57fb2c8b22def5d2e7c4690d124cd6e524461fc9c68d962da7"
Jul 07 03:22:06.609612 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:06.609584    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 07 03:22:06.613815 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:06.613788    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 07 03:22:07.980094 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:07.980062    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="183633a3-721d-488b-8a68-1812ea1b2621" path="/var/lib/kubelet/pods/183633a3-721d-488b-8a68-1812ea1b2621/volumes"
Jul 07 03:22:14.331261 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331229    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"]
Jul 07 03:22:14.331673 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331561    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="storage-initializer"
Jul 07 03:22:14.331673 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331574    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="storage-initializer"
Jul 07 03:22:14.331673 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331594    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main"
Jul 07 03:22:14.331673 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331599    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main"
Jul 07 03:22:14.331673 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331607    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="183633a3-721d-488b-8a68-1812ea1b2621" containerName="storage-initializer"
Jul 07 03:22:14.331673 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331612    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="183633a3-721d-488b-8a68-1812ea1b2621" containerName="storage-initializer"
Jul 07 03:22:14.331673 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331624    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="183633a3-721d-488b-8a68-1812ea1b2621" containerName="main"
Jul 07 03:22:14.331673 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331630    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="183633a3-721d-488b-8a68-1812ea1b2621" containerName="main"
Jul 07 03:22:14.332102 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331683    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3038731a-e281-4b3f-a1db-aace2be0ddc5" containerName="main"
Jul 07 03:22:14.332102 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.331690    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="183633a3-721d-488b-8a68-1812ea1b2621" containerName="main"
Jul 07 03:22:14.336524 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.336503    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.339865 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.339844    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-9nvvn\""
Jul 07 03:22:14.339990 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.339879    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 07 03:22:14.344318 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.344297    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"]
Jul 07 03:22:14.511748 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.511717    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfk7w\" (UniqueName: \"kubernetes.io/projected/10da08d6-7c36-4bb3-9c73-e305e752d283-kube-api-access-rfk7w\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.511944 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.511763    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.511944 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.511831    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.511944 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.511876    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10da08d6-7c36-4bb3-9c73-e305e752d283-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.512058 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.511949    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.512094 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.512054    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.570460 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.570423    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"]
Jul 07 03:22:14.573920 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.573901    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:14.576288 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.576267    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-inline-config-test-epp-sa-dockercfg-7629k\""
Jul 07 03:22:14.583719 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.583661    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"]
Jul 07 03:22:14.613169 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.613139    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfk7w\" (UniqueName: \"kubernetes.io/projected/10da08d6-7c36-4bb3-9c73-e305e752d283-kube-api-access-rfk7w\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.613169 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.613176    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.613430 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.613227    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.613430 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.613254    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10da08d6-7c36-4bb3-9c73-e305e752d283-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.613430 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.613312    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.613430 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.613352    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.613666 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.613643    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-tmp-dir\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.613722 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.613681    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-model-cache\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.613722 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.613705    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-home\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.615643 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.615617    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-dshm\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.615980 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.615964    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10da08d6-7c36-4bb3-9c73-e305e752d283-tls-certs\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.620828 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.620807    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfk7w\" (UniqueName: \"kubernetes.io/projected/10da08d6-7c36-4bb3-9c73-e305e752d283-kube-api-access-rfk7w\") pod \"scheduler-inline-config-test-kserve-7c6c58cc96-94kmg\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.646833 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.646796    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:14.714509 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.714474    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2qqn\" (UniqueName: \"kubernetes.io/projected/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-kube-api-access-l2qqn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:14.714663 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.714558    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:14.714663 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.714589    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:14.782224 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.782168    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"]
Jul 07 03:22:14.785989 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:22:14.785959    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10da08d6_7c36_4bb3_9c73_e305e752d283.slice/crio-aa1785173675b3aa509e6f588acc5482a31d10c4c00ce632b15b368f114f8bd2 WatchSource:0}: Error finding container aa1785173675b3aa509e6f588acc5482a31d10c4c00ce632b15b368f114f8bd2: Status 404 returned error can't find the container with id aa1785173675b3aa509e6f588acc5482a31d10c4c00ce632b15b368f114f8bd2
Jul 07 03:22:14.815280 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.815252    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:14.815419 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.815292    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:14.815419 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.815365    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2qqn\" (UniqueName: \"kubernetes.io/projected/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-kube-api-access-l2qqn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:14.815819 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.815770    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tokenizer-uds\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:14.817941 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.817920    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:14.824345 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.824324    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2qqn\" (UniqueName: \"kubernetes.io/projected/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-kube-api-access-l2qqn\") pod \"scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") " pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:14.884315 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:14.884280    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:15.044250 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:15.044223    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"]
Jul 07 03:22:15.045840 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:22:15.045812    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f40815_b48d_4c72_b8bb_4d2e3a8a670e.slice/crio-8c0793a0e289a1fc6bc17c1fbb67b2b34e96882e970b391c62c59fe57e53b9da WatchSource:0}: Error finding container 8c0793a0e289a1fc6bc17c1fbb67b2b34e96882e970b391c62c59fe57e53b9da: Status 404 returned error can't find the container with id 8c0793a0e289a1fc6bc17c1fbb67b2b34e96882e970b391c62c59fe57e53b9da
Jul 07 03:22:15.636798 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:15.636763    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg" event={"ID":"10da08d6-7c36-4bb3-9c73-e305e752d283","Type":"ContainerStarted","Data":"c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf"}
Jul 07 03:22:15.637211 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:15.636820    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg" event={"ID":"10da08d6-7c36-4bb3-9c73-e305e752d283","Type":"ContainerStarted","Data":"aa1785173675b3aa509e6f588acc5482a31d10c4c00ce632b15b368f114f8bd2"}
Jul 07 03:22:15.638261 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:15.638239    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr" event={"ID":"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e","Type":"ContainerStarted","Data":"4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397"}
Jul 07 03:22:15.638360 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:15.638277    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr" event={"ID":"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e","Type":"ContainerStarted","Data":"8c0793a0e289a1fc6bc17c1fbb67b2b34e96882e970b391c62c59fe57e53b9da"}
Jul 07 03:22:15.638360 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:15.638339    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:22:15.647553 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:15.647532    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:15.654035 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:15.653995    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg" podStartSLOduration=1.65398081 podStartE2EDuration="1.65398081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:22:15.653013133 +0000 UTC m=+3848.296173656" watchObservedRunningTime="2026-07-07 03:22:15.65398081 +0000 UTC m=+3848.297141332"
Jul 07 03:22:15.667404 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:15.667361    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr" podStartSLOduration=1.667347264 podStartE2EDuration="1.667347264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:22:15.667055403 +0000 UTC m=+3848.310215937" watchObservedRunningTime="2026-07-07 03:22:15.667347264 +0000 UTC m=+3848.310507786"
Jul 07 03:22:24.647693 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:24.647646    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:24.660145 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:24.660118    2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:24.694578 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:24.694540    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:22:46.644991 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:22:46.644957    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:23:08.175280 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:08.175251    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:23:08.184110 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:08.184075    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:23:13.349154 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.349120    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"]
Jul 07 03:23:13.356376 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.356340    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"]
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:23:13.359237    2543 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]:  > containerID="4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397" cmd=["/bin/sleep","15"]
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.359290    2543 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr" message="" err=<
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]:  >
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:23:13.359301    2543 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]:         , stdout: , stderr: , exit code -1
Jul 07 03:23:13.359330 ip-10-0-132-101 kubenswrapper[2543]:  > pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr" podUID="f1f40815-b48d-4c72-b8bb-4d2e3a8a670e" containerName="main" containerID="cri-o://4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397"
Jul 07 03:23:13.360036 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.359347    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr" podUID="f1f40815-b48d-4c72-b8bb-4d2e3a8a670e" containerName="main" containerID="cri-o://4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397" gracePeriod=60
Jul 07 03:23:13.614203 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.614164    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:23:13.631727 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.631687    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tls-certs\") pod \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") "
Jul 07 03:23:13.631894 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.631803    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tokenizer-uds\") pod \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") "
Jul 07 03:23:13.631894 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.631840    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2qqn\" (UniqueName: \"kubernetes.io/projected/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-kube-api-access-l2qqn\") pod \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\" (UID: \"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e\") "
Jul 07 03:23:13.632071 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.632049    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "f1f40815-b48d-4c72-b8bb-4d2e3a8a670e" (UID: "f1f40815-b48d-4c72-b8bb-4d2e3a8a670e"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:23:13.634431 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.634402    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-kube-api-access-l2qqn" (OuterVolumeSpecName: "kube-api-access-l2qqn") pod "f1f40815-b48d-4c72-b8bb-4d2e3a8a670e" (UID: "f1f40815-b48d-4c72-b8bb-4d2e3a8a670e"). InnerVolumeSpecName "kube-api-access-l2qqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:23:13.634431 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.634409    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1f40815-b48d-4c72-b8bb-4d2e3a8a670e" (UID: "f1f40815-b48d-4c72-b8bb-4d2e3a8a670e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:23:13.733469 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.733431    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2qqn\" (UniqueName: \"kubernetes.io/projected/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-kube-api-access-l2qqn\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:23:13.733469 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.733462    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:23:13.733469 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.733472    2543 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e-tokenizer-uds\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:23:13.899820 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.899784    2543 generic.go:356] "Generic (PLEG): container finished" podID="f1f40815-b48d-4c72-b8bb-4d2e3a8a670e" containerID="4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397" exitCode=0
Jul 07 03:23:13.900013 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.899834    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"
Jul 07 03:23:13.900013 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.899883    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr" event={"ID":"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e","Type":"ContainerDied","Data":"4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397"}
Jul 07 03:23:13.900013 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.899933    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr" event={"ID":"f1f40815-b48d-4c72-b8bb-4d2e3a8a670e","Type":"ContainerDied","Data":"8c0793a0e289a1fc6bc17c1fbb67b2b34e96882e970b391c62c59fe57e53b9da"}
Jul 07 03:23:13.900013 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.899955    2543 scope.go:117] "RemoveContainer" containerID="4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397"
Jul 07 03:23:13.910394 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.910376    2543 scope.go:117] "RemoveContainer" containerID="4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397"
Jul 07 03:23:13.910673 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:23:13.910655    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397\": container with ID starting with 4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397 not found: ID does not exist" containerID="4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397"
Jul 07 03:23:13.910726 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.910681    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397"} err="failed to get container status \"4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397\": rpc error: code = NotFound desc = could not find container \"4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397\": container with ID starting with 4ba75bc1072020abbcef85626fcb76e848902c46111f0d85c6e05c2bad1b8397 not found: ID does not exist"
Jul 07 03:23:13.921844 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.921814    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"]
Jul 07 03:23:13.925435 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.925414    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-router-scheduler-d9ddfb42pr"]
Jul 07 03:23:13.980667 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:13.980635    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1f40815-b48d-4c72-b8bb-4d2e3a8a670e" path="/var/lib/kubelet/pods/f1f40815-b48d-4c72-b8bb-4d2e3a8a670e/volumes"
Jul 07 03:23:28.383528 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.383464    2543 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg" podUID="10da08d6-7c36-4bb3-9c73-e305e752d283" containerName="main" containerID="cri-o://c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf" gracePeriod=45
Jul 07 03:23:28.641023 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.640961    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:23:28.765465 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765418    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-home\") pod \"10da08d6-7c36-4bb3-9c73-e305e752d283\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") "
Jul 07 03:23:28.765667 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765476    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10da08d6-7c36-4bb3-9c73-e305e752d283-tls-certs\") pod \"10da08d6-7c36-4bb3-9c73-e305e752d283\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") "
Jul 07 03:23:28.765667 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765526    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-dshm\") pod \"10da08d6-7c36-4bb3-9c73-e305e752d283\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") "
Jul 07 03:23:28.765667 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765556    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-model-cache\") pod \"10da08d6-7c36-4bb3-9c73-e305e752d283\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") "
Jul 07 03:23:28.765667 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765635    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfk7w\" (UniqueName: \"kubernetes.io/projected/10da08d6-7c36-4bb3-9c73-e305e752d283-kube-api-access-rfk7w\") pod \"10da08d6-7c36-4bb3-9c73-e305e752d283\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") "
Jul 07 03:23:28.765876 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765669    2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-tmp-dir\") pod \"10da08d6-7c36-4bb3-9c73-e305e752d283\" (UID: \"10da08d6-7c36-4bb3-9c73-e305e752d283\") "
Jul 07 03:23:28.765876 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765724    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-home" (OuterVolumeSpecName: "home") pod "10da08d6-7c36-4bb3-9c73-e305e752d283" (UID: "10da08d6-7c36-4bb3-9c73-e305e752d283"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:23:28.765876 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765854    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-model-cache" (OuterVolumeSpecName: "model-cache") pod "10da08d6-7c36-4bb3-9c73-e305e752d283" (UID: "10da08d6-7c36-4bb3-9c73-e305e752d283"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:23:28.766007 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765957    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "10da08d6-7c36-4bb3-9c73-e305e752d283" (UID: "10da08d6-7c36-4bb3-9c73-e305e752d283"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:23:28.766007 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765973    2543 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-home\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:23:28.766007 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.765990    2543 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-model-cache\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:23:28.767875 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.767844    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-dshm" (OuterVolumeSpecName: "dshm") pod "10da08d6-7c36-4bb3-9c73-e305e752d283" (UID: "10da08d6-7c36-4bb3-9c73-e305e752d283"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 07 03:23:28.767875 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.767842    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10da08d6-7c36-4bb3-9c73-e305e752d283-kube-api-access-rfk7w" (OuterVolumeSpecName: "kube-api-access-rfk7w") pod "10da08d6-7c36-4bb3-9c73-e305e752d283" (UID: "10da08d6-7c36-4bb3-9c73-e305e752d283"). InnerVolumeSpecName "kube-api-access-rfk7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 07 03:23:28.768071 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.767906    2543 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10da08d6-7c36-4bb3-9c73-e305e752d283-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "10da08d6-7c36-4bb3-9c73-e305e752d283" (UID: "10da08d6-7c36-4bb3-9c73-e305e752d283"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 07 03:23:28.867307 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.867271    2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfk7w\" (UniqueName: \"kubernetes.io/projected/10da08d6-7c36-4bb3-9c73-e305e752d283-kube-api-access-rfk7w\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:23:28.867510 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.867338    2543 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-tmp-dir\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:23:28.867510 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.867357    2543 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10da08d6-7c36-4bb3-9c73-e305e752d283-tls-certs\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:23:28.867510 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.867371    2543 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10da08d6-7c36-4bb3-9c73-e305e752d283-dshm\") on node \"ip-10-0-132-101.ec2.internal\" DevicePath \"\""
Jul 07 03:23:28.964301 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.964182    2543 generic.go:356] "Generic (PLEG): container finished" podID="10da08d6-7c36-4bb3-9c73-e305e752d283" containerID="c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf" exitCode=0
Jul 07 03:23:28.964301 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.964248    2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"
Jul 07 03:23:28.964301 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.964278    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg" event={"ID":"10da08d6-7c36-4bb3-9c73-e305e752d283","Type":"ContainerDied","Data":"c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf"}
Jul 07 03:23:28.964547 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.964334    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg" event={"ID":"10da08d6-7c36-4bb3-9c73-e305e752d283","Type":"ContainerDied","Data":"aa1785173675b3aa509e6f588acc5482a31d10c4c00ce632b15b368f114f8bd2"}
Jul 07 03:23:28.964547 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.964356    2543 scope.go:117] "RemoveContainer" containerID="c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf"
Jul 07 03:23:28.973137 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.973113    2543 scope.go:117] "RemoveContainer" containerID="c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf"
Jul 07 03:23:28.973430 ip-10-0-132-101 kubenswrapper[2543]: E0707 03:23:28.973413    2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf\": container with ID starting with c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf not found: ID does not exist" containerID="c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf"
Jul 07 03:23:28.973490 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.973439    2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf"} err="failed to get container status \"c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf\": rpc error: code = NotFound desc = could not find container \"c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf\": container with ID starting with c7ddb79e09119847041f0da966b7a243f5851be5f747b117d585bc4be6d46dbf not found: ID does not exist"
Jul 07 03:23:28.984814 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.984790    2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"]
Jul 07 03:23:28.990036 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:28.990013    2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-inline-config-test-kserve-7c6c58cc96-94kmg"]
Jul 07 03:23:29.979506 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:23:29.979463    2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10da08d6-7c36-4bb3-9c73-e305e752d283" path="/var/lib/kubelet/pods/10da08d6-7c36-4bb3-9c73-e305e752d283/volumes"
Jul 07 03:24:03.962716 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:03.962638    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:05.355029 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:05.354990    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:06.761201 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:06.761162    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:08.183181 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:08.183146    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:09.570326 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:09.570297    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:10.964782 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:10.964756    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:12.389850 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:12.389816    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:13.816475 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:13.816447    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:15.280565 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:15.280535    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:16.753123 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:16.753097    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:18.168252 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:18.168223    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:19.546403 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:19.546368    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:20.924301 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:20.924273    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:22.314225 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:22.314171    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:23.687889 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:23.687858    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:25.099892 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:25.099861    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-bt7g2_b5ede82e-7e84-438e-81df-f2ff0deb2257/istio-proxy/0.log"
Jul 07 03:24:28.904278 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:28.904243    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-kl894_99f96f1a-b75a-4956-a8ab-ca32a63f6d26/authorino/0.log"
Jul 07 03:24:28.944573 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:28.944547    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-g9hbm_a1177dde-fbad-4ec5-a787-dcdd8cdbe812/kuadrant-console-plugin/0.log"
Jul 07 03:24:31.636231 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.636179    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8t88/must-gather-zd6kb"]
Jul 07 03:24:31.636700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.636539    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10da08d6-7c36-4bb3-9c73-e305e752d283" containerName="main"
Jul 07 03:24:31.636700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.636552    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="10da08d6-7c36-4bb3-9c73-e305e752d283" containerName="main"
Jul 07 03:24:31.636700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.636570    2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1f40815-b48d-4c72-b8bb-4d2e3a8a670e" containerName="main"
Jul 07 03:24:31.636700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.636576    2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1f40815-b48d-4c72-b8bb-4d2e3a8a670e" containerName="main"
Jul 07 03:24:31.636700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.636628    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1f40815-b48d-4c72-b8bb-4d2e3a8a670e" containerName="main"
Jul 07 03:24:31.636700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.636638    2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="10da08d6-7c36-4bb3-9c73-e305e752d283" containerName="main"
Jul 07 03:24:31.639657 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.639636    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8t88/must-gather-zd6kb"
Jul 07 03:24:31.642568 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.642552    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8t88\"/\"kube-root-ca.crt\""
Jul 07 03:24:31.643574 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.643555    2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j8t88\"/\"default-dockercfg-s74vw\""
Jul 07 03:24:31.643687 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.643557    2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j8t88\"/\"openshift-service-ca.crt\""
Jul 07 03:24:31.649347 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.649325    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8t88/must-gather-zd6kb"]
Jul 07 03:24:31.713735 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.713697    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-748l2\" (UniqueName: \"kubernetes.io/projected/e04ab5bf-f058-49a0-bdf1-6ee817fc5a61-kube-api-access-748l2\") pod \"must-gather-zd6kb\" (UID: \"e04ab5bf-f058-49a0-bdf1-6ee817fc5a61\") " pod="openshift-must-gather-j8t88/must-gather-zd6kb"
Jul 07 03:24:31.713928 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.713829    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e04ab5bf-f058-49a0-bdf1-6ee817fc5a61-must-gather-output\") pod \"must-gather-zd6kb\" (UID: \"e04ab5bf-f058-49a0-bdf1-6ee817fc5a61\") " pod="openshift-must-gather-j8t88/must-gather-zd6kb"
Jul 07 03:24:31.815139 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.815099    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e04ab5bf-f058-49a0-bdf1-6ee817fc5a61-must-gather-output\") pod \"must-gather-zd6kb\" (UID: \"e04ab5bf-f058-49a0-bdf1-6ee817fc5a61\") " pod="openshift-must-gather-j8t88/must-gather-zd6kb"
Jul 07 03:24:31.815139 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.815141    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-748l2\" (UniqueName: \"kubernetes.io/projected/e04ab5bf-f058-49a0-bdf1-6ee817fc5a61-kube-api-access-748l2\") pod \"must-gather-zd6kb\" (UID: \"e04ab5bf-f058-49a0-bdf1-6ee817fc5a61\") " pod="openshift-must-gather-j8t88/must-gather-zd6kb"
Jul 07 03:24:31.815514 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.815492    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e04ab5bf-f058-49a0-bdf1-6ee817fc5a61-must-gather-output\") pod \"must-gather-zd6kb\" (UID: \"e04ab5bf-f058-49a0-bdf1-6ee817fc5a61\") " pod="openshift-must-gather-j8t88/must-gather-zd6kb"
Jul 07 03:24:31.822917 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.822888    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-748l2\" (UniqueName: \"kubernetes.io/projected/e04ab5bf-f058-49a0-bdf1-6ee817fc5a61-kube-api-access-748l2\") pod \"must-gather-zd6kb\" (UID: \"e04ab5bf-f058-49a0-bdf1-6ee817fc5a61\") " pod="openshift-must-gather-j8t88/must-gather-zd6kb"
Jul 07 03:24:31.949339 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:31.949254    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8t88/must-gather-zd6kb"
Jul 07 03:24:32.079832 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:32.079739    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8t88/must-gather-zd6kb"]
Jul 07 03:24:32.082247 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:24:32.082213    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode04ab5bf_f058_49a0_bdf1_6ee817fc5a61.slice/crio-18740be1ba7d615bd829bd970e96630b2ab2e316f20f20b4a913597860b3ac25 WatchSource:0}: Error finding container 18740be1ba7d615bd829bd970e96630b2ab2e316f20f20b4a913597860b3ac25: Status 404 returned error can't find the container with id 18740be1ba7d615bd829bd970e96630b2ab2e316f20f20b4a913597860b3ac25
Jul 07 03:24:32.084550 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:32.084536    2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 07 03:24:32.245824 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:32.245734    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8t88/must-gather-zd6kb" event={"ID":"e04ab5bf-f058-49a0-bdf1-6ee817fc5a61","Type":"ContainerStarted","Data":"18740be1ba7d615bd829bd970e96630b2ab2e316f20f20b4a913597860b3ac25"}
Jul 07 03:24:33.252141 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:33.252045    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8t88/must-gather-zd6kb" event={"ID":"e04ab5bf-f058-49a0-bdf1-6ee817fc5a61","Type":"ContainerStarted","Data":"50d0aa3096d5757a8e09eb9be7ca23c06d00a96290d729fb40da60bd04e8d7b9"}
Jul 07 03:24:33.252609 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:33.252271    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8t88/must-gather-zd6kb" event={"ID":"e04ab5bf-f058-49a0-bdf1-6ee817fc5a61","Type":"ContainerStarted","Data":"2499fd6ac26641591474bb761c10631237d6673cd27f4090c52655239a5f4663"}
Jul 07 03:24:34.433159 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:34.433132    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-drrrz_dfd04d15-d522-4e36-83ff-1631d6f859aa/global-pull-secret-syncer/0.log"
Jul 07 03:24:34.616144 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:34.616111    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vkx8c_6fb4b23e-61ae-449e-abad-c58f2c9f5bc6/konnectivity-agent/0.log"
Jul 07 03:24:34.638426 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:34.638354    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-101.ec2.internal_36ceb552a78de1cb2a021e04da828958/haproxy/0.log"
Jul 07 03:24:39.665211 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:39.664226    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-kl894_99f96f1a-b75a-4956-a8ab-ca32a63f6d26/authorino/0.log"
Jul 07 03:24:39.786020 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:39.785980    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-g9hbm_a1177dde-fbad-4ec5-a787-dcdd8cdbe812/kuadrant-console-plugin/0.log"
Jul 07 03:24:41.753098 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:41.753058    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc25k_cc39428c-c98a-46b6-ad0d-ba35deb34614/node-exporter/0.log"
Jul 07 03:24:41.776338 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:41.776311    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc25k_cc39428c-c98a-46b6-ad0d-ba35deb34614/kube-rbac-proxy/0.log"
Jul 07 03:24:41.796880 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:41.796845    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mc25k_cc39428c-c98a-46b6-ad0d-ba35deb34614/init-textfile/0.log"
Jul 07 03:24:41.822955 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:41.822926    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-cbxjp_cf095b1e-ae19-46b7-a5bc-24828d33d3e9/kube-rbac-proxy-main/0.log"
Jul 07 03:24:41.844213 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:41.844158    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-cbxjp_cf095b1e-ae19-46b7-a5bc-24828d33d3e9/kube-rbac-proxy-self/0.log"
Jul 07 03:24:41.865449 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:41.865378    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-cbxjp_cf095b1e-ae19-46b7-a5bc-24828d33d3e9/openshift-state-metrics/0.log"
Jul 07 03:24:42.148992 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:42.148943    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-6jbst_15733967-d83f-416c-aae2-603f1f91cfa4/prometheus-operator-admission-webhook/0.log"
Jul 07 03:24:42.190890 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:42.190858    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-567b4cc688-4j2sv_b63caec1-c0c2-4f6c-b210-f1940d7cfff3/telemeter-client/0.log"
Jul 07 03:24:42.222136 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:42.222102    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-567b4cc688-4j2sv_b63caec1-c0c2-4f6c-b210-f1940d7cfff3/reload/0.log"
Jul 07 03:24:42.247550 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:42.247517    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-567b4cc688-4j2sv_b63caec1-c0c2-4f6c-b210-f1940d7cfff3/kube-rbac-proxy/0.log"
Jul 07 03:24:44.202387 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.202358    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/1.log"
Jul 07 03:24:44.209214 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.209164    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-chdqk_6c294fd4-a99a-4441-9f26-b636c3b01562/console-operator/2.log"
Jul 07 03:24:44.653334 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.653307    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-zcxv2_bc96cf09-5eda-4783-87a7-a2c97120b7ff/download-server/0.log"
Jul 07 03:24:44.928535 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.928397    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8t88/must-gather-zd6kb" podStartSLOduration=13.038359894 podStartE2EDuration="13.928371852s" totalImagesPullingTime="890.011958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-07 03:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:24:33.283754411 +0000 UTC m=+3985.926914934" watchObservedRunningTime="2026-07-07 03:24:44.928371852 +0000 UTC m=+3997.571532370"
Jul 07 03:24:44.930247 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.930217    2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"]
Jul 07 03:24:44.936212 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.936167    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:44.940867 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.940823    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"]
Jul 07 03:24:44.960866 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.960832    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-sys\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:44.961113 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.961095    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-podres\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:44.961301 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.961285    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-proc\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:44.961443 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.961425    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-lib-modules\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:44.961604 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:44.961581    2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w8gc\" (UniqueName: \"kubernetes.io/projected/dae7f748-e3ea-4f88-8286-cc181c04fd96-kube-api-access-7w8gc\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.063058 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.063018    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-sys\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.063463 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.063440    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-podres\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.063773 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.063754    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-proc\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.063988 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.063972    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-lib-modules\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.064285 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.064267    2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w8gc\" (UniqueName: \"kubernetes.io/projected/dae7f748-e3ea-4f88-8286-cc181c04fd96-kube-api-access-7w8gc\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.064754 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.063924    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-proc\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.064869 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.063688    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-podres\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.064869 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.064165    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-lib-modules\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.064869 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.063386    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dae7f748-e3ea-4f88-8286-cc181c04fd96-sys\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.072653 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.072624    2543 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w8gc\" (UniqueName: \"kubernetes.io/projected/dae7f748-e3ea-4f88-8286-cc181c04fd96-kube-api-access-7w8gc\") pod \"perf-node-gather-daemonset-89llf\" (UID: \"dae7f748-e3ea-4f88-8286-cc181c04fd96\") " pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.248779 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.248675    2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:45.430142 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:45.430103    2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"]
Jul 07 03:24:45.432997 ip-10-0-132-101 kubenswrapper[2543]: W0707 03:24:45.432963    2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddae7f748_e3ea_4f88_8286_cc181c04fd96.slice/crio-ae7d1679c9e6cd7f254ccb554e62d14babc3c8fc52bec7af14643e383911ea34 WatchSource:0}: Error finding container ae7d1679c9e6cd7f254ccb554e62d14babc3c8fc52bec7af14643e383911ea34: Status 404 returned error can't find the container with id ae7d1679c9e6cd7f254ccb554e62d14babc3c8fc52bec7af14643e383911ea34
Jul 07 03:24:46.248700 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:46.248678    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dzjkl_5921583f-ac66-4059-80f0-0cf81c329d20/dns/0.log"
Jul 07 03:24:46.268850 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:46.268821    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dzjkl_5921583f-ac66-4059-80f0-0cf81c329d20/kube-rbac-proxy/0.log"
Jul 07 03:24:46.365508 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:46.365470    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf" event={"ID":"dae7f748-e3ea-4f88-8286-cc181c04fd96","Type":"ContainerStarted","Data":"465db214eea479e165d42edc802d37bcc33edd1de0d8689daac1c9d771a94d78"}
Jul 07 03:24:46.365696 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:46.365533    2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf" event={"ID":"dae7f748-e3ea-4f88-8286-cc181c04fd96","Type":"ContainerStarted","Data":"ae7d1679c9e6cd7f254ccb554e62d14babc3c8fc52bec7af14643e383911ea34"}
Jul 07 03:24:46.365696 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:46.365563    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:46.374854 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:46.374822    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nqvp2_e052eaa2-eca7-4894-8be6-e6b273851fa7/dns-node-resolver/0.log"
Jul 07 03:24:46.381626 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:46.381581    2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf" podStartSLOduration=2.3815648400000002 podStartE2EDuration="2.38156484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-07 03:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-07 03:24:46.380310127 +0000 UTC m=+3999.023470651" watchObservedRunningTime="2026-07-07 03:24:46.38156484 +0000 UTC m=+3999.024725362"
Jul 07 03:24:46.876489 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:46.876450    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-55d6c7f4cb-fvv4h_24978d7d-4b78-4416-97ea-712c6a147898/registry/0.log"
Jul 07 03:24:46.898493 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:46.898460    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6l9zx_c703c034-4b57-4d22-a865-01ab46b75790/node-ca/0.log"
Jul 07 03:24:48.655323 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:48.655298    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5hrxw_25136846-eadf-4a17-93c9-76e0191971ff/serve-healthcheck-canary/0.log"
Jul 07 03:24:49.222406 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:49.222377    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pkpzw_013a90cb-6fb8-4e1f-974d-04ea1d5078dc/kube-rbac-proxy/0.log"
Jul 07 03:24:49.246297 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:49.246264    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pkpzw_013a90cb-6fb8-4e1f-974d-04ea1d5078dc/exporter/0.log"
Jul 07 03:24:49.266472 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:49.266444    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pkpzw_013a90cb-6fb8-4e1f-974d-04ea1d5078dc/extractor/0.log"
Jul 07 03:24:52.056735 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:52.056697    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d5cdd5cf7-f6h48_f8ee2e5f-89d3-4a05-bf24-fd3c79fccba7/manager/0.log"
Jul 07 03:24:52.080610 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:52.080581    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-xkhwz_4e33ee66-cd16-488f-a6e1-b332256dbb52/openshift-lws-operator/0.log"
Jul 07 03:24:52.386856 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:52.385932    2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-j8t88/perf-node-gather-daemonset-89llf"
Jul 07 03:24:52.654858 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:52.654783    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-f8f7445f4-2dtqc_26d8d45b-7585-4e82-ae5a-bc00dac45dba/manager/0.log"
Jul 07 03:24:53.714461 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:24:53.714429    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-m5wwv_be4a725e-d246-44d1-ad23-8c7648e7ae7a/seaweedfs/0.log"
Jul 07 03:25:00.197213 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:00.197151    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sdd9x_bca64acd-88bf-4c2e-9677-50add32ca830/kube-multus-additional-cni-plugins/0.log"
Jul 07 03:25:00.217938 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:00.217914    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sdd9x_bca64acd-88bf-4c2e-9677-50add32ca830/egress-router-binary-copy/0.log"
Jul 07 03:25:00.237089 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:00.237064    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sdd9x_bca64acd-88bf-4c2e-9677-50add32ca830/cni-plugins/0.log"
Jul 07 03:25:00.255497 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:00.255474    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sdd9x_bca64acd-88bf-4c2e-9677-50add32ca830/bond-cni-plugin/0.log"
Jul 07 03:25:00.273993 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:00.273970    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sdd9x_bca64acd-88bf-4c2e-9677-50add32ca830/routeoverride-cni/0.log"
Jul 07 03:25:00.293739 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:00.293710    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sdd9x_bca64acd-88bf-4c2e-9677-50add32ca830/whereabouts-cni-bincopy/0.log"
Jul 07 03:25:00.312661 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:00.312634    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sdd9x_bca64acd-88bf-4c2e-9677-50add32ca830/whereabouts-cni/0.log"
Jul 07 03:25:00.499949 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:00.499872    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-m7vbn_5cf5689f-ebb4-4e70-bce5-6680ea8dcd3e/kube-multus/0.log"
Jul 07 03:25:00.679251 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:00.679223    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wllmf_50534db7-9f97-4088-bc8a-b33de25d0d36/network-metrics-daemon/0.log"
Jul 07 03:25:00.697533 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:00.697487    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wllmf_50534db7-9f97-4088-bc8a-b33de25d0d36/kube-rbac-proxy/0.log"
Jul 07 03:25:01.474382 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:01.474339    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gh2mp_de69c094-35c2-4b5a-8d70-efdc95adec5b/ovn-controller/0.log"
Jul 07 03:25:01.489237 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:01.489213    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gh2mp_de69c094-35c2-4b5a-8d70-efdc95adec5b/ovn-acl-logging/0.log"
Jul 07 03:25:01.514459 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:01.514435    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gh2mp_de69c094-35c2-4b5a-8d70-efdc95adec5b/kube-rbac-proxy-node/0.log"
Jul 07 03:25:01.533074 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:01.533039    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gh2mp_de69c094-35c2-4b5a-8d70-efdc95adec5b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 07 03:25:01.549373 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:01.549343    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gh2mp_de69c094-35c2-4b5a-8d70-efdc95adec5b/northd/0.log"
Jul 07 03:25:01.568921 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:01.568895    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gh2mp_de69c094-35c2-4b5a-8d70-efdc95adec5b/nbdb/0.log"
Jul 07 03:25:01.588971 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:01.588949    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gh2mp_de69c094-35c2-4b5a-8d70-efdc95adec5b/sbdb/0.log"
Jul 07 03:25:01.648895 ip-10-0-132-101 kubenswrapper[2543]: I0707 03:25:01.648866    2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gh2mp_de69c094-35c2-4b5a-8d70-efdc95adec5b/ovnkube-controller/0.log"